|
When we discuss with clients, we hear that they need to have the ability to harness insights from information with a purpose to make well timed, impactful, and actionable enterprise selections. A typical sample with data-driven organizations is that they’ve many different information sources they should ingest into their analytics methods. This requires them to construct handbook information pipelines spanning throughout their operational databases, information lakes, streaming information, and information inside their warehouse. As a consequence of this complicated setup, it could take information engineers weeks and even months to construct information ingestion pipelines. These information pipelines are pricey, and the delays can result in missed enterprise alternatives. Additionally, information warehouses are more and more turning into mission crucial methods that require excessive availability, reliability, and safety.
Amazon Redshift is a totally managed petabyte-scale information warehouse utilized by tens of hundreds of shoppers to simply, shortly, securely, and cost-effectively analyze all their information at any scale. This 12 months at re:Invent, Amazon Redshift has introduced various options that can assist you simplify information ingestion and get to insights simply and shortly, inside a safe, dependable surroundings.
In this weblog, I introduce a few of these new options that fit into two primary classes:
- Simplify information ingestion
- Amazon Redshift now helps auto-copy from Amazon S3 (obtainable in preview). With this new functionality, Amazon Redshift robotically hundreds the information that arrive in an Amazon Simple Storage Service (Amazon S3) location that you simply specify into your information warehouse. The information can use any of the codecs supported by the Amazon Redshift copy command, corresponding to CSV, JSON, Parquet, and Avro. In this fashion, you don’t must manually or repeatedly run copy procedures. Amazon Redshift automates file ingestion and takes care of data-loading steps beneath the hood.
- With Amazon Aurora zero-ETL integration with Amazon Redshift, you should utilize Amazon Redshift for close to real-time analytics and machine studying on petabytes of transactional information saved on Amazon Aurora MySQL databases (obtainable in restricted preview). With this functionality, you may select the Amazon Aurora databases containing the info you wish to analyze with Amazon Redshift. Data is then replicated into your information warehouse inside seconds after transactional information is written into Amazon Aurora, eliminating the necessity to construct and keep complicated information pipelines. You can replicate information from a number of Amazon Aurora databases into the identical Amazon Redshift occasion to run analytics throughout a number of purposes. With close to real-time entry to transactional information, you may leverage Amazon Redshift’s analytics and capabilities, corresponding to built-in machine studying (ML), materialized views, information sharing, and federated entry to a number of information shops and information lakes, to derive insights from transactional and different information.
- With the final availability of Amazon Redshift Streaming Ingestion, now you can natively ingest a whole lot of megabytes of information per second from Amazon Kinesis Data Streams and Amazon MSK into an Amazon Redshift materialized view and question it in seconds. Learn extra in this publish.
- Make your information warehouse safer and dependable
- You can now enhance the provision of your information warehouse by selecting a number of Availability Zone (AZ) deployments. Multi-AZ deployments to your Amazon Redshift clusters can be found in preview and cut back restoration instances to seconds via automated restoration. In this fashion, you may construct options which are extra compliant with the suggestions of the Reliability Pillar of the AWS Well-Architected Framework.
- With dynamic information masking (obtainable in preview), you may shield delicate info saved in your information warehouse and be sure that solely the related information is accessible by customers based mostly on their roles. You can restrict how a lot identifiable information is seen to customers utilizing a number of ranges of insurance policies so completely different customers and teams can have completely different ranges of information entry with out having to create a number of copies of information. Dynamic information masking enhances different granular entry management capabilities in Amazon Redshift together with row-level and column-level safety and role-based entry controls. In this fashion, Dynamic Data Masking helps you meet necessities for GDPR, CCPA, and different privateness rules.
- Amazon Redshift now helps central entry controls for information sharing with AWS Lake Formation (obtainable in public preview). You can now use Lake Formation to simplify governance of information shared from Amazon Redshift and centrally handle granular entry throughout all data-sharing customers.
There have been different attention-grabbing information for Amazon Redshift at re:Invent you may need already heard about:
- The common availability of Amazon Redshift integration for Apache Spark makes it straightforward to construct and run Spark purposes on Amazon Redshift and Redshift Serverless, opening up the info warehouse for a broader set of AWS analytics and machine studying options.
- AWS Backup now helps Amazon Redshift. AWS Backup permits you to outline a central backup coverage to handle information safety of your purposes and also can shield your Amazon Redshift clusters. In this fashion, you’ve got a constant expertise when managing information safety throughout all supported companies.
Availability and Pricing
Multi-AZ deployments, central entry management for information sharing with AWS Lake Formation, auto-copy from Amazon S3, and dynamic information masking can be found in preview in US East (Ohio), US East (N. Virginia), US West (Oregon), Asia Pacific (Tokyo), Europe (Ireland), and Europe (Stockholm).
There is not any extra price for utilizing auto-copy from Amazon S3 and close to real-time analytics on transactional information. There is not any additional cost for dynamic information masking and central entry management for information sharing. For extra info, see Amazon Redshift pricing.
These new capabilities take you one step additional in analyzing all of your information throughout information sources with easy information ingestion capabilities, whereas enhancing the safety and reliability of your information warehouse.
— Danilo