Content
Over the past few weeks, we have introduced some changes to how the Apify API gets records from key-value stores and uploads them. These changes improve the API’s stability and usability. While they will mostly happen under the hood, they are relevant to any developer using Apify.
Main changes:
- Adjustments to how data are stored
- Unlimited file size when uploading to key-value store
- Key-value store requests served from Apify servers without redirects to S3
- Deprecation of Direct Upload API endpoint
Adjustments to how data are stored
Until now, every record uploaded using the Put record endpoint was compressed using gzip before uploading. This has changed. Records are now stored in the state you upload them. This means it is up to you to decide whether the record is stored compressed or uncompressed.
You can compress a record and use the Content-Encoding request header to let our platform know which compression format it uses. We recommend compressing large key-value records to save storage space and network traffic.
If you use Apify SDK or our JavaScript API client, your files are compressed automatically by default. We recommend using the JavaScript API client, which compresses your data before they are sent to our servers and decompresses them when you retrieve them. This keeps your storage costs as low as possible.
Unlimited file size when uploading to key-value store
To simplify the upload process, we have removed the 9MB-per-file upload limit for key-value stores. You can now upload a file of any size when using the Put record API endpoint.
Previously, unlimited size uploads were only available when using the Direct upload endpoint. This means you no longer need to call the Direct upload endpoint, get an upload URL, and then send your file to that URL, leaving you more time for important things.
Key-value store requests are served from Apify servers without redirects to S3
Requests to the Get record endpoint are no longer redirected to AWS S3. It was confusing, and we found that most customers disabled this feature using the disableRedirect parameter.
From now, all records in key-value stores are served directly from Apify servers, without redirects. Thus, we removed the disableRedirect parameter as it no longer has any effect. While requests using disableRedirect will still work, we recommend not using the parameter.
Deprecation of Direct Upload API endpoint
This endpoint is no longer necessary following the change that allows files of unlimited size to be uploaded using the Put record endpoint. The endpoint is still available and existing integrations will still work; however, we recommend migrating to the Put record endpoint.
If you know someone who will find these updates useful, please share them. We are continuously working on updates and improvements, so keep an eye out for our newsletter every month and for new blog posts on how to use the Apify platform.
