|
Every 12 months on March 14 (3.14), AWS Pi Day highlights AWS improvements that assist you handle and work along with your information. What began in 2021 as a strategy to commemorate the fifteenth launch anniversary of Amazon Simple Storage Service (Amazon S3) has now grown into an occasion that highlights how cloud applied sciences are remodeling information administration, analytics, and AI.
This 12 months, AWS Pi Day returns with a give attention to accelerating analytics and AI innovation with a unified information basis on AWS. The information panorama is present process a profound transformation as AI emerges in most enterprise methods, with analytics and AI workloads more and more converging round a variety of the identical information and workflows. You want a straightforward strategy to entry all of your information and use all of your most well-liked analytics and AI instruments in a single built-in expertise. This AWS Pi Day, we’re introducing a slate of recent capabilities that assist you construct unified and built-in information experiences.
The subsequent technology of Amazon SageMaker: The middle of all of your information, analytics, and AI
At re:Invent 2024, we launched the following technology of Amazon SageMaker, the middle of all of your information, analytics, and AI. SageMaker contains nearly all of the parts you want for information exploration, preparation and integration, large information processing, quick SQL analytics, machine studying (ML) mannequin growth and coaching, and generative AI software growth. With this new technology of Amazon SageMaker, SageMaker Lakehouse gives you with unified entry to your information and SageMaker Catalog lets you meet your governance and safety necessities. You can learn the launch weblog submit written by my colleague Antje to study extra particulars.
Core to the following technology of Amazon SageMaker is SageMaker Unified Studio, a single information and AI growth surroundings the place you should use all of your information and instruments for analytics and AI. SageMaker Unified Studio is now usually obtainable.
SageMaker Unified Studio facilitates collaboration amongst information scientists, analysts, engineers, and builders as they work on information, analytics, AI workflows, and purposes. It gives acquainted instruments from AWS analytics and artificial intelligence and machine studying (AI/ML) companies, together with information processing, SQL analytics, ML mannequin growth, and generative AI software growth, right into a single person expertise. To discover the advantages of SageMaker Unified Studio, learn Accelerate analytics and AI innovation with the following technology of Amazon SageMaker, by analytics leaders at AWS.
SageMaker Unified Studio additionally brings chosen capabilities from Amazon Bedrock into SageMaker. You can now quickly prototype, customise, and share generative AI purposes utilizing foundation fashions (FMs) and superior options akin to Amazon Bedrock Knowledge Bases, Amazon Bedrock Guardrails, Amazon Bedrock Agents, and Amazon Bedrock Flows to create tailor-made options aligned along with your necessities and accountable AI pointers all inside SageMaker.
Last however not least, Amazon Q Developer is now usually obtainable in SageMaker Unified Studio. Amazon Q Developer gives generative AI powered help for information and AI growth. It helps you with duties like writing SQL queries, constructing extract, remodel, and cargo (ETL) jobs, and troubleshooting, and is out there in the Free tier and Pro tier for current subscribers.
You can study extra in regards to the normal availability of SageMaker Unified Studio on this latest weblog submit written by my colleague Donnie.
During re:Invent 2024, we additionally launched Amazon SageMaker Lakehouse as a part of the following technology of SageMaker. SageMaker Lakehouse unifies all of your information throughout Amazon S3 information lakes, Amazon Redshift information warehouses, and third-party and federated information sources. It helps you construct highly effective analytics and AI/ML purposes on a single copy of your information. SageMaker Lakehouse provides you the flexibleness to entry and question your information in-place with Apache Iceberg–appropriate instruments and engines. In addition, zero-ETL integrations automate the method of bringing information into SageMaker Lakehouse from AWS information sources akin to Amazon Aurora or Amazon DynamoDB and from purposes akin to Salesforce, Facebook Ads, Instagram Ads, ServiceNow, SAP, Zendesk, and Zoho CRM. The full checklist of integrations is out there within the SageMaker Lakehouse FAQ.
Building an information basis with Amazon S3
Building an information basis is the cornerstone of accelerating analytics and AI workloads, enabling organizations to seamlessly handle, uncover, and make the most of their information belongings at any scale. Amazon S3 is the world’s finest place to construct an information lake, with nearly limitless scale, and it gives the important basis for this transformation.
I’m at all times astonished to study in regards to the scale at which we function Amazon S3: It at the moment holds over 400 trillion objects, exabytes of knowledge, and processes a mind-blowing 150 million requests per second. Just a decade in the past, not even 100 clients had been storing greater than a petabyte (PB) of knowledge on S3. Today, hundreds of shoppers have surpassed the 1 PB milestone.
Amazon S3 shops exabytes of tabular information, and it averages over 15 million requests to tabular information per second. To assist you cut back the undifferentiated heavy lifting when managing your tabular information in S3 buckets, we introduced Amazon S3 Tables at AWS re:Invent 2024. S3 Tables are the primary cloud object retailer with built-in assist for Apache Iceberg. S3 tables are particularly optimized for analytics workloads, leading to as much as threefold quicker question throughput and as much as tenfold larger transactions per second in comparison with self-managed tables.
Today, we’re saying the normal availability of Amazon S3 Tables integration with Amazon SageMaker Lakehouse Amazon S3 Tables now combine with Amazon SageMaker Lakehouse, making it straightforward so that you can entry S3 Tables from AWS analytics companies akin to Amazon Redshift, Amazon Athena, Amazon EMR, AWS Glue, and Apache Iceberg–appropriate engines akin to Apache Spark or PyIceberg. SageMaker Lakehouse allows centralized administration of fine-grained information entry permissions for S3 Tables and different sources and persistently applies them throughout all engines.
For these of you who use a third-party catalog, have a customized catalog implementation, or solely want fundamental learn and write entry to tabular information in a single desk bucket, we’ve added new APIs which are appropriate with the Iceberg REST Catalog commonplace. This allows any Iceberg-compatible software to seamlessly create, replace, checklist, and delete tables in an S3 desk bucket. For unified information administration throughout your whole tabular information, information governance, and fine-grained entry controls, you may also use S3 Tables with SageMaker Lakehouse.
To assist you entry S3 Tables, we’ve launched updates within the AWS Management Console. You can now create a desk, populate it with information, and question it straight from the S3 console utilizing Amazon Athena, making it simpler to get began and analyze information in S3 desk buckets.
The following screenshot exhibits the way to entry Athena straight from the S3 console.
When I choose Query tables with Athena or Create desk with Athena, it opens the Athena console on the right information supply, catalog, and database.
Since re:Invent 2024, we’ve continued so as to add new capabilities to S3 Tables at a speedy tempo. For instance, we added schema definition assist to the CreateDesk API and now you can create as much as 10,000 tables in an S3 desk bucket. We additionally launched S3 Tables into eight extra AWS Regions, with the newest being Asia Pacific (Seoul, Singapore, Sydney) on March 4, with extra to return. You can seek advice from the S3 Tables AWS Regions web page of the documentation to get the checklist of the eleven Regions the place S3 Tables can be found at present.
Amazon S3 Metadata—introduced throughout re:Invent 2024— has been usually obtainable since January 27. It’s the quickest and easiest method that will help you uncover and perceive your S3 information with automated, effortlessly-queried metadata that updates in close to actual time. S3 Metadata works with S3 object tags. Tags assist you logically group information for a wide range of causes, akin to to use IAM insurance policies to offer fine-grained entry, specify tag-based filters to handle object lifecycle guidelines, and selectively replicate information to a different Region. In Regions the place S3 Metadata is out there, you may seize and question customized metadata that’s saved as object tags. To cut back the price related to object tags when utilizing S3 Metadata, Amazon S3 diminished pricing for S3 object tagging by 35 % in all Regions, making it cheaper to make use of customized metadata.
AWS Pi Day 2025
Over the years, AWS Pi Day has showcased main milestones in cloud storage and information analytics. This 12 months, the AWS Pi Day digital occasion will characteristic a spread of matters designed for builders and technical decision-makers, information engineers, AI/ML practitioners, and IT leaders. Key highlights embrace deep dives, reside demos, and professional classes on all of the companies and capabilities I mentioned on this submit.
By attending this occasion, you’ll study how one can speed up your analytics and AI innovation. You’ll find out how you should use S3 Tables with native Apache Iceberg assist and S3 Metadata to construct scalable information lakes that serve each conventional analytics and rising AI/ML workloads. You’ll additionally uncover the following technology of Amazon SageMaker, the middle for all of your information, analytics, and AI, to assist your groups collaborate and construct quicker from a unified studio, utilizing acquainted AWS instruments with entry to all of your information whether or not it’s saved in information lakes, information warehouses, or third-party or federated information sources.
For these seeking to keep forward of the most recent cloud developments, AWS Pi Day 2025 is an occasion you may’t miss. Whether you’re constructing information lakehouses, coaching AI fashions, constructing generative AI purposes, or optimizing analytics workloads, the insights shared will assist you maximize the worth of your information.
Tune in at present and discover the most recent in cloud information innovation. Don’t miss the chance to have interaction with AWS consultants, companions, and clients shaping the way forward for information, analytics, and AI.
If you missed the digital occasion on March 14, you may go to the occasion web page at any time—we’ll hold all of the content material obtainable on-demand there!
3/18/2025: Added hyperlink to Big Data weblog.
How is the News Blog doing? Take this 1 minute survey!
(This survey is hosted by an exterior firm. AWS handles your data as described within the AWS Privacy Notice. AWS will personal the info gathered through this survey and won’t share the knowledge collected with survey respondents.)