From 54cb633750acd7d1cb06e897ec82e49be4083e1a Mon Sep 17 00:00:00 2001 From: Quentin Date: Sun, 21 Jul 2024 17:30:23 +0200 Subject: [PATCH 01/10] Work in progress Blossom BUD-05 --- buds/02.md | 22 +++++++++++++ buds/05.md | 95 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 117 insertions(+) create mode 100644 buds/05.md diff --git a/buds/02.md b/buds/02.md index e645943..7ee377d 100644 --- a/buds/02.md +++ b/buds/02.md @@ -19,6 +19,28 @@ A blob descriptor is a JSON object containing `url`, `sha256`, `size`, `type`, a - `size` The size of the blob in bytes - `type` (optional) The MIME type of the blob - `uploaded` The unix timestamp of when the blob was uploaded to the server +- `compressed` (optional) A JSON object containing the `sha256`, `size`, `library`, `version`, and `parameters` JSON object containing the compression parameters `quality` and `mode`. This field is optional and should only be included if the blob was compressed + +Blob Descriptor example: +```json +{ + "sha256": "b1674191a88ec5cdd733e4240a81803105dc412d6c6708d53ab94fc248f4f553", + "uploaded": 1708773959, + "size": 123456, + "type": "application/pdf", + "compressed": { + "sha256": "e2e2e1d2b9c04e2a9914c245e3f789d230e3c2d5e3a0f5c6e4b4e3f9c7d4f3e2", + "size": 56789, + "library": "brotli", + "version": "1.0.7", + "parameters": { + "quality": 11, + "mode": "text" + } + } +} + +``` Servers may include additional fields in the descriptor like `magnet`, `infohash`, or `ipfs` depending on other protocols they support diff --git a/buds/05.md b/buds/05.md new file mode 100644 index 0000000..5d9ef80 --- /dev/null +++ b/buds/05.md @@ -0,0 +1,95 @@ +BUD-05 +====== + +Deterministic File Compression +--------------- + +`draft` `optional` + +## Overview + +This document outlines the implementation of a deterministic file compression method for Blossom, specifying the metadata that must be stored for each compressed blob and the endpoint for retrieving this metadata. + +## Server Adaptation + +File storage servers `MUST` store the following metadata for each compressed blob: + +- File hash (sha256) +- Uploaded timestamp (uploaded) +- Original file size (size) +- Original file type (type) +- Compressed file hash (compressed.sha256) +- Compressed file size (compressed.size) +- Compression library (compression.library) +- Compression library version (compression.version) +- Compression library parameters (compression.parameters) + +Servers `MUST` normalize the blob metadata to ensure that the same compressed blob with the same parameters will always produce the same result, thereby ensuring deterministic compression and anonymizing any potential user data. + +## GET /metadata - Retrieve blob metadata + +The GET /metadata/ endpoint `MUST` return a [Blob Descriptor](https://github.com/hzrd149/blossom/blob/master/buds/02.md#blob-descriptor) containing the metadata fields for the requested blob or an error object if the blob does not exist. + +The endpoint `MUST` accept an optional file extension in the URL. ie. `.pdf`, `.png`, etc + +Example response: + +```json +{ + "url": "cdn.nostrcheck.me/b1674191a88ec5cdd733e4240a81803105dc412d6c6708d53ab94fc248f4f553", + "sha256": "b1674191a88ec5cdd733e4240a81803105dc412d6c6708d53ab94fc248f4f553", + "uploaded": 1708773959, + "size": 123456, + "type": "application/pdf", + "compressed": { + "sha256": "e2e2e1d2b9c04e2a9914c245e3f789d230e3c2d5e3a0f5c6e4b4e3f9c7d4f3e2", + "size": 56789, + "library": "brotli", + "version": "1.0.7", + "parameters": { + "quality": 11, + "mode": "text" + } + } +} + +``` + +## Example Flow + +1. Client signs authorization event and uploads blob to Server A with 'compress' field set to true +1. Server A compresses the blob using it's preferred library and parameters +1. Server A stores the blob metadata +1. Server A returns [Blob Descriptor](./02.md#blob-descriptor) +1. (Optional) Client verify the metadata matches the original blob compressing and comparing the blob locally with the same returned parameters +1. Client uploads the compressed blob to Server B using the original authorization event and the 'compress' field set to false. + +1. Server B verifies blob hash metadata matches `x` tag in authorization event +1. Server B stores the blob metadata +1. Server B returns [Blob Descriptor](./02.md#blob-descriptor) +1. (Optional) Client verify the metadata matches the original blob compressing the blob locally with the same parameters + +(optional using the /mirror endpoint) + +1. Client sends the `url` to Server C `/mirror` using the original authorization event +1. Server C downloads blob from Server A or B using the url field +1. Server C verifies downloaded blob hash matches `x` tag in authorization event +1. Server C verify the metadata matches the original blob compressing the blob locally with the same parameters +1. Server C returns [Blob Descriptor](./02.md#blob-descriptor) + + + +## Accepted Libraries, Versions, and Parameters + +The following libraries, versions, and parameters are accepted for deterministic compression: + +(This is just a DEMO table, the actual table will be updated with the final list of libraries, versions, and parameters) + +| Library | Version | Quality Range | Modes | +|---------|---------|---------------|------------------------| +| Brotli | 1.0.7 | 0-11 | text, font, generic | +| Gzip | 1.10 | 0-9 | text, font, generic | +| Zstd | 1.4.5 | 1-22 | text, font, generic | +| Lz4 | 1.9.2 | 1-12 | text, font, generic | + + From 8c93401037a461fef551a7364608ecf3877d2d8d Mon Sep 17 00:00:00 2001 From: Quentin Date: Sun, 21 Jul 2024 17:34:46 +0200 Subject: [PATCH 02/10] some changes --- buds/05.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/buds/05.md b/buds/05.md index 5d9ef80..1918c3c 100644 --- a/buds/05.md +++ b/buds/05.md @@ -26,6 +26,10 @@ File storage servers `MUST` store the following metadata for each compressed blo Servers `MUST` normalize the blob metadata to ensure that the same compressed blob with the same parameters will always produce the same result, thereby ensuring deterministic compression and anonymizing any potential user data. +## Client adaptation + +Clients `MUST` verify the metadata of the compressed blob to ensure that the compressed blob matches the original blob. Clients `MUST` sign and publish a new authorization event with the new blob hash. Clients `MUST` upload the compressed blob to the server using the new authorization event and the 'compress' field set to false or not set. + ## GET /metadata - Retrieve blob metadata The GET /metadata/ endpoint `MUST` return a [Blob Descriptor](https://github.com/hzrd149/blossom/blob/master/buds/02.md#blob-descriptor) containing the metadata fields for the requested blob or an error object if the blob does not exist. @@ -61,13 +65,12 @@ Example response: 1. Server A compresses the blob using it's preferred library and parameters 1. Server A stores the blob metadata 1. Server A returns [Blob Descriptor](./02.md#blob-descriptor) -1. (Optional) Client verify the metadata matches the original blob compressing and comparing the blob locally with the same returned parameters -1. Client uploads the compressed blob to Server B using the original authorization event and the 'compress' field set to false. - -1. Server B verifies blob hash metadata matches `x` tag in authorization event +1. Client verify the metadata matches the original blob compressing and comparing the blob locally with the same returned parameters +1. Client signs and publishes a new authorization event with the new blob hash and the 'compress' field set to false +1. Client uploads the compressed blob to Server B using the new authorization event and the 'compress' field set to false or not set +1. Server B verifies blob hash metadata matches `x` tag in the new authorization event 1. Server B stores the blob metadata 1. Server B returns [Blob Descriptor](./02.md#blob-descriptor) -1. (Optional) Client verify the metadata matches the original blob compressing the blob locally with the same parameters (optional using the /mirror endpoint) From ba4eb9b8f93843a3a698f1a6666f81c7b9fb1845 Mon Sep 17 00:00:00 2001 From: Quentin Date: Sun, 21 Jul 2024 17:35:57 +0200 Subject: [PATCH 03/10] fix typo --- buds/05.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/buds/05.md b/buds/05.md index 1918c3c..5051dde 100644 --- a/buds/05.md +++ b/buds/05.md @@ -74,10 +74,9 @@ Example response: (optional using the /mirror endpoint) -1. Client sends the `url` to Server C `/mirror` using the original authorization event +1. Client sends the `url` to Server C `/mirror` using the compressed authorization event 1. Server C downloads blob from Server A or B using the url field -1. Server C verifies downloaded blob hash matches `x` tag in authorization event -1. Server C verify the metadata matches the original blob compressing the blob locally with the same parameters +1. Server C verifies downloaded blob hash matches `x` tag in authorization event (using sha256 or compressed.sha256) 1. Server C returns [Blob Descriptor](./02.md#blob-descriptor) From 3b6de3071930e4e01ac482fc0e51cf37c85301bb Mon Sep 17 00:00:00 2001 From: Quentin Date: Thu, 22 Aug 2024 09:51:23 +0200 Subject: [PATCH 04/10] reorder branches. sync with source repo --- buds/01.md | 8 ++--- buds/02.md | 30 +++-------------- buds/03.md | 78 +++++++++++++++++++++++++++++++++++++++++++ buds/05.md | 97 ------------------------------------------------------ 4 files changed, 86 insertions(+), 127 deletions(-) create mode 100644 buds/03.md delete mode 100644 buds/05.md diff --git a/buds/01.md b/buds/01.md index 25bd4ba..a9c3016 100644 --- a/buds/01.md +++ b/buds/01.md @@ -60,7 +60,7 @@ Authorization: Nostr eyJpZCI6IjhlY2JkY2RkNTMyOTIwMDEwNTUyNGExNDI4NzkxMzg4MWIzOWQ ## Endpoints -All endpoints MUST be served from the root path (eg. `https://cdn.example.com/upload`, etc). This allows clients to talk to servers interchangeably when uploading or reteriving blobs +All endpoints MUST be served from the root path (eg. `https://cdn.example.com/upload`, etc). This allows clients to talk to servers interchangeably when uploading or retrieving blobs ## Error Responses @@ -92,7 +92,7 @@ This ensures that if a user was to copy or reuse the redirect URL it would still ### Get Authorization (optional) -The server may optionally require authorization when reteriving blobs from the `GET /` endpoint +The server may optionally require authorization when retrieving blobs from the `GET /` endpoint In this case the server MUST perform additional checks on the authorization event @@ -101,7 +101,7 @@ In this case the server MUST perform additional checks on the authorization even If the client did not send an `Authorization` header the server must respond with the appropriate HTTP status code `401` (Unauthorized) -Example event for retreiving a single blob: +Example event for retrieving a single blob: ```json { @@ -119,7 +119,7 @@ Example event for retreiving a single blob: } ``` -Example event for retreiving multiple blobs from single server: +Example event for retrieving multiple blobs from single server: ```json { diff --git a/buds/02.md b/buds/02.md index 7ee377d..b9afce6 100644 --- a/buds/02.md +++ b/buds/02.md @@ -8,7 +8,7 @@ Blob upload and management _All pubkeys MUST be in hex format_ -Defines the `/upload`, `/list` and `DELETE /` enpoints +Defines the `/upload`, `/list` and `DELETE /` endpoints ## Blob Descriptor @@ -19,28 +19,6 @@ A blob descriptor is a JSON object containing `url`, `sha256`, `size`, `type`, a - `size` The size of the blob in bytes - `type` (optional) The MIME type of the blob - `uploaded` The unix timestamp of when the blob was uploaded to the server -- `compressed` (optional) A JSON object containing the `sha256`, `size`, `library`, `version`, and `parameters` JSON object containing the compression parameters `quality` and `mode`. This field is optional and should only be included if the blob was compressed - -Blob Descriptor example: -```json -{ - "sha256": "b1674191a88ec5cdd733e4240a81803105dc412d6c6708d53ab94fc248f4f553", - "uploaded": 1708773959, - "size": 123456, - "type": "application/pdf", - "compressed": { - "sha256": "e2e2e1d2b9c04e2a9914c245e3f789d230e3c2d5e3a0f5c6e4b4e3f9c7d4f3e2", - "size": 56789, - "library": "brotli", - "version": "1.0.7", - "parameters": { - "quality": 11, - "mode": "text" - } - } -} - -``` Servers may include additional fields in the descriptor like `magnet`, `infohash`, or `ipfs` depending on other protocols they support @@ -83,7 +61,7 @@ Example Authorization event: The `/list/` endpoint MUST return a JSON array of [Blob Descriptor](#blob-descriptor) that where uploaded by the specified pubkey -The endpoint MUST support a `since` and `until` query parameter to limit the returned blobs by thier `uploaded` date +The endpoint MUST support a `since` and `until` query parameter to limit the returned blobs by their `uploaded` date Servers may reject a list for any reason and MUST respond with the appropriate HTTP `4xx` status code and an error message explaining the reason for the rejection @@ -116,7 +94,7 @@ Example Authorization event: Servers MUST accept `DELETE` requests to the `/` endpoint -Servers may reject a delete request for any reason and should respond with the aproperate HTTP `4xx` status code and an error message explaining the reason for the rejection +Servers may reject a delete request for any reason and should respond with the appropriate HTTP `4xx` status code and an error message explaining the reason for the rejection ### Delete Authorization (required) @@ -138,7 +116,7 @@ Example Authorization event: "created_at": 1708774469, "tags": [ ["t", "delete"], - ["x", "b1674191a88ec5cdd733e4240a81803105dc412d6c6708d53ab94fc248f4f553 "], + ["x", "b1674191a88ec5cdd733e4240a81803105dc412d6c6708d53ab94fc248f4f553"], ["expiration", "1708858680"] ], "sig": "2ba9af680505583e3eb289a1624a08661a2f6fa2e5566a5ee0036333d517f965e0ffba7f5f7a57c2de37e00a2e85fd7999076468e52bdbcfad8abb76b37a94b0" diff --git a/buds/03.md b/buds/03.md new file mode 100644 index 0000000..0cf971a --- /dev/null +++ b/buds/03.md @@ -0,0 +1,78 @@ +BUD-03 +====== + +User Server List +------------------------- + +`draft` `optional` + +Defines a replaceable event using `kind:10063` to advertise the blossom servers a user uses to host their blobs. + +The event MUST include at least one `server` tag containing the full server URL including the `http://` or `https://`. + +The order of these tags is important and should be arranged with the users most "reliable" or "trusted" servers being first. + +The `.content` field is not used. + +```json +{ + "id": "e4bee088334cb5d38cff1616e964369c37b6081be997962ab289d6c671975d71", + "pubkey": "781208004e09102d7da3b7345e64fd193cd1bc3fce8fdae6008d77f9cabcd036", + "content": "", + "kind": 10063, + "created_at": 1708774162, + "tags": [ + ["server", "https://cdn.self.hosted"], + ["server", "https://cdn.satellite.earth"] + ], + "sig": "cc5efa74f59e80622c77cacf4dd62076bcb7581b45e9acff471e7963a1f4d8b3406adab5ee1ac9673487480e57d20e523428e60ffcc7e7a904ac882cfccfc653" +} +``` + +## Client Upload Implementation + +When uploading blobs clients MUST attempt to upload the blob to at least the first `server` listed in the users server list. + +Optionally clients MAY upload the blob to all the servers or mirror the blob to the other servers if they support [BUD-04](./04.md) + +This ensures that the blob is available in multiple locations in the case one of the servers goes offline. + +## Client Retrieval Implementation + +When extracting the SHA256 hash from the URL clients MUST use the last occurrence of a 64 char hex string. This allows clients to extract hashes from blossom URLs and SOME non-blossom URLs. + +In all the following examples, the hash `b1674191a88ec5cdd733e4240a81803105dc412d6c6708d53ab94fc248f4f553` should be selected + +- Blossom URLs + - `https://blossom.example.com/b1674191a88ec5cdd733e4240a81803105dc412d6c6708d53ab94fc248f4f553.pdf` + - `https://cdn.example.com/b1674191a88ec5cdd733e4240a81803105dc412d6c6708d53ab94fc248f4f553` +- Non Blossom URLs + - `https://cdn.example.com/user/ec4425ff5e9446080d2f70440188e3ca5d6da8713db7bdeef73d0ed54d9093f0/media/b1674191a88ec5cdd733e4240a81803105dc412d6c6708d53ab94fc248f4f553.pdf` + - `https://cdn.example.com/media/user-name/documents/b1674191a88ec5cdd733e4240a81803105dc412d6c6708d53ab94fc248f4f553.pdf` + - `http://download.example.com/downloads/b1674191a88ec5cdd733e4240a81803105dc412d6c6708d53ab94fc248f4f553` + - `http://media.example.com/documents/b1/67/b1674191a88ec5cdd733e4240a81803105dc412d6c6708d53ab94fc248f4f553.pdf` + +In the context of nostr events, clients SHOULD use the author's server list when looking for blobs that are no longer available at the original URL. + +Take the following event as an example + +```json +{ + "id": "834185269f4ab72539193105060dbb1c8b2efd702d14481cea345c47beefe6eb", + "pubkey": "ec4425ff5e9446080d2f70440188e3ca5d6da8713db7bdeef73d0ed54d9093f0", + "content": "I've developed a new open source P2P e-cash system called Bitcoin. check it out\nhttps://cdn.broken-domain.com/b1674191a88ec5cdd733e4240a81803105dc412d6c6708d53ab94fc248f4f553.pdf", + "kind": 1, + "created_at": 1297484820, + "tags": [], + "sig": "bd4bb200bdd5f7ffe5dbc3e539052e27b05d6f9f528e255b1bc4261cc16b8f2ad85c89eef990c5f2eee756ef71b4c571ecf6a88ad12f7338e321dd60c6a903b5" +} +``` + +Once the client discovers that the URL `https://cdn.broken-domain.com/b1674191a88ec5cdd733e4240a81803105dc412d6c6708d53ab94fc248f4f553.pdf` is no longer available. It can perform the following steps to find the blob: + +1. Get the SHA256 has from the URL +2. Look for the authors server list `kind:10063` +3. If found, Attempt to retrieve the blob from each `server` listed started with the first +3. If not found, the client MAY fallback to using a well-known popular blossom server to retrieve the blob + +This ensures clients can quickly find missing blobs using the users list of trusted servers. diff --git a/buds/05.md b/buds/05.md deleted file mode 100644 index 5051dde..0000000 --- a/buds/05.md +++ /dev/null @@ -1,97 +0,0 @@ -BUD-05 -====== - -Deterministic File Compression ---------------- - -`draft` `optional` - -## Overview - -This document outlines the implementation of a deterministic file compression method for Blossom, specifying the metadata that must be stored for each compressed blob and the endpoint for retrieving this metadata. - -## Server Adaptation - -File storage servers `MUST` store the following metadata for each compressed blob: - -- File hash (sha256) -- Uploaded timestamp (uploaded) -- Original file size (size) -- Original file type (type) -- Compressed file hash (compressed.sha256) -- Compressed file size (compressed.size) -- Compression library (compression.library) -- Compression library version (compression.version) -- Compression library parameters (compression.parameters) - -Servers `MUST` normalize the blob metadata to ensure that the same compressed blob with the same parameters will always produce the same result, thereby ensuring deterministic compression and anonymizing any potential user data. - -## Client adaptation - -Clients `MUST` verify the metadata of the compressed blob to ensure that the compressed blob matches the original blob. Clients `MUST` sign and publish a new authorization event with the new blob hash. Clients `MUST` upload the compressed blob to the server using the new authorization event and the 'compress' field set to false or not set. - -## GET /metadata - Retrieve blob metadata - -The GET /metadata/ endpoint `MUST` return a [Blob Descriptor](https://github.com/hzrd149/blossom/blob/master/buds/02.md#blob-descriptor) containing the metadata fields for the requested blob or an error object if the blob does not exist. - -The endpoint `MUST` accept an optional file extension in the URL. ie. `.pdf`, `.png`, etc - -Example response: - -```json -{ - "url": "cdn.nostrcheck.me/b1674191a88ec5cdd733e4240a81803105dc412d6c6708d53ab94fc248f4f553", - "sha256": "b1674191a88ec5cdd733e4240a81803105dc412d6c6708d53ab94fc248f4f553", - "uploaded": 1708773959, - "size": 123456, - "type": "application/pdf", - "compressed": { - "sha256": "e2e2e1d2b9c04e2a9914c245e3f789d230e3c2d5e3a0f5c6e4b4e3f9c7d4f3e2", - "size": 56789, - "library": "brotli", - "version": "1.0.7", - "parameters": { - "quality": 11, - "mode": "text" - } - } -} - -``` - -## Example Flow - -1. Client signs authorization event and uploads blob to Server A with 'compress' field set to true -1. Server A compresses the blob using it's preferred library and parameters -1. Server A stores the blob metadata -1. Server A returns [Blob Descriptor](./02.md#blob-descriptor) -1. Client verify the metadata matches the original blob compressing and comparing the blob locally with the same returned parameters -1. Client signs and publishes a new authorization event with the new blob hash and the 'compress' field set to false -1. Client uploads the compressed blob to Server B using the new authorization event and the 'compress' field set to false or not set -1. Server B verifies blob hash metadata matches `x` tag in the new authorization event -1. Server B stores the blob metadata -1. Server B returns [Blob Descriptor](./02.md#blob-descriptor) - -(optional using the /mirror endpoint) - -1. Client sends the `url` to Server C `/mirror` using the compressed authorization event -1. Server C downloads blob from Server A or B using the url field -1. Server C verifies downloaded blob hash matches `x` tag in authorization event (using sha256 or compressed.sha256) -1. Server C returns [Blob Descriptor](./02.md#blob-descriptor) - - - -## Accepted Libraries, Versions, and Parameters - -The following libraries, versions, and parameters are accepted for deterministic compression: - -(This is just a DEMO table, the actual table will be updated with the final list of libraries, versions, and parameters) - -| Library | Version | Quality Range | Modes | -|---------|---------|---------------|------------------------| -| Brotli | 1.0.7 | 0-11 | text, font, generic | -| Gzip | 1.10 | 0-9 | text, font, generic | -| Zstd | 1.4.5 | 1-22 | text, font, generic | -| Lz4 | 1.9.2 | 1-12 | text, font, generic | - - From 45709636481827c31426d5a23055599427e50b05 Mon Sep 17 00:00:00 2001 From: Quentin Date: Tue, 10 Sep 2024 10:23:10 +0200 Subject: [PATCH 05/10] BUD06 - Upload requirements --- README.md | 3 ++- buds/06.md | 55 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 57 insertions(+), 1 deletion(-) create mode 100644 buds/06.md diff --git a/README.md b/README.md index 9c9c85c..701180d 100644 --- a/README.md +++ b/README.md @@ -16,9 +16,10 @@ Blossom Servers expose four endpoints for managing blobs - `GET /` (optional file `.ext`) [BUD-01](./buds/01.md#get-sha256---get-blob) - `HEAD /` (optional file `.ext`) [BUD-01](./buds/01.md#head-sha256---has-blob) -- `PUT /upload` [BUD-2](./buds/02.md#put-upload---upload-blob) +- `PUT /upload` [BUD-02](./buds/02.md#put-upload---upload-blob) - `Authentication`: Signed [nostr event](./buds/02.md#upload-authorization-required) - Return a blob descriptor +- `HEAD /upload` [BUD-06](./buds/06.md#head-upload---upload-requirements) - `GET /list/` [BUD-02](./buds/02.md#get-listpubkey---list-blobs) - Returns an array of blob descriptors - `Authentication` _(optional)_: Signed [nostr event](./buds/02.md#list-authorization-optional) diff --git a/buds/06.md b/buds/06.md new file mode 100644 index 0000000..e79b418 --- /dev/null +++ b/buds/06.md @@ -0,0 +1,55 @@ +BUD-06 +====== + +Upload requirements +--------------- + +`draft` `optional` + +Defines how clients can verify if the upload can be completed before sending the blob to the server. This mechanism helps prevent unnecessary traffic to other endpoints by rejecting files based on their hash, size, MIME type or other server-specific requirements. + +## HEAD /upload - Upload requirements + +The `HEAD /upload` endpoint `MUST` use the `Content-Digest`, `X-Content-Type` and `X-Content-Length` headers sent by client to get the SHA-256 hash, MIME type and size of the blob that will be uploaded, returning a HTTP status code and a custom header `X-Upload-Message` to indicate some human readable message about the upload requirements. + +### Headers + +- `Content-Digest`: A string formatted as `sha-256=::`, where `` is the base64-encoded SHA-256 hash of the blob. +- `X-Content-Length`: An integer that represents the blob size in bytes. +- `X-Content-Type`: A string that specifies the fblobile's MIME type, like `application/pdf` or `image/png`. +- `X-Upload-Message`: A human readable message that explains the reason why the upload cannot proceed. + +### Examples + +Example request from the client: + +```http +X-Content-Type: application/pdf +X-Content-Length: 184292 +Content-Digest: SHA-256=:88a74d0b866c8ba79251a11fe5ac807839226870e77355f02eaf68b156522576: +``` + +Example response from the server if the upload can be done: + +```http +HTTP/1.1 200 OK +``` + +If the upload cannot proceed, the server `MUST` return an appropriate HTTP status code and a custom header `X-Upload-Message` with a human readable error message. + +Some examples of error messages: + +```http +HTTP/1.1 400 Bad Request +X-Upload-Message: Invalid Content-Digest header format. Expected format: sha-256=:: +``` + +```http +HTTP/1.1 413 Content Too Large +X-Upload-Message: File too large. Max allowed size is 100MB +``` + +```http +HTTP/1.1 415 Unsupported Media Type +X-Upload-Message: Unsupported file type. +``` \ No newline at end of file From 00a378d81f1b90445797dc7c56f93c3679d73c24 Mon Sep 17 00:00:00 2001 From: Quentin Date: Tue, 10 Sep 2024 11:34:25 +0200 Subject: [PATCH 06/10] Add BUD-06 to readme --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 701180d..72ed62e 100644 --- a/README.md +++ b/README.md @@ -40,6 +40,7 @@ See the [BUDs](./buds) folder and specifically [BUD-01](./buds/01.md) and [BUD-0 - [BUD-02: Blob upload and management](./buds/02.md) - [BUD-03: User Server List](./buds/03.md) - [BUD-04: Mirroring blobs](./buds/04.md) +- [BUD-06: Upload requirements](./buds/06.md) ## Event kinds From 5593e3d38938684ed7281b72915d7d38e1e63c2c Mon Sep 17 00:00:00 2001 From: Quentin <125748180+quentintaranpino@users.noreply.github.com> Date: Thu, 12 Sep 2024 09:30:55 +0200 Subject: [PATCH 07/10] Update buds/06.md Co-authored-by: hzrd149 <8001706+hzrd149@users.noreply.github.com> --- buds/06.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/buds/06.md b/buds/06.md index e79b418..a3bbe16 100644 --- a/buds/06.md +++ b/buds/06.md @@ -35,7 +35,7 @@ Example response from the server if the upload can be done: HTTP/1.1 200 OK ``` -If the upload cannot proceed, the server `MUST` return an appropriate HTTP status code and a custom header `X-Upload-Message` with a human readable error message. +If the upload cannot proceed, the server `MUST` return an appropriate `4xx` HTTP status code and a custom header `X-Upload-Message` with a human readable error message. Some examples of error messages: From 86d8da48703d5e9bbea464c682f34bb591e8759d Mon Sep 17 00:00:00 2001 From: quentintaranpino Date: Thu, 12 Sep 2024 09:35:07 +0200 Subject: [PATCH 08/10] upload authorization. More examples, fix some typo --- buds/06.md | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/buds/06.md b/buds/06.md index e79b418..d302095 100644 --- a/buds/06.md +++ b/buds/06.md @@ -19,6 +19,12 @@ The `HEAD /upload` endpoint `MUST` use the `Content-Digest`, `X-Content-Type` an - `X-Content-Type`: A string that specifies the fblobile's MIME type, like `application/pdf` or `image/png`. - `X-Upload-Message`: A human readable message that explains the reason why the upload cannot proceed. +### Upload Authorization + +The `HEAD /upload` endpoint MAY accept an `upload` authorization event using the `Authorization` header similar to what is used in the [`PUT /upload`](./02.md#upload-authorization-required) endpoint + +If the server requires authorization to upload it may respond with the `401` status code, or if authorization was provided and is invalid or not permitted it may respond with `403` status code + ### Examples Example request from the client: @@ -35,7 +41,8 @@ Example response from the server if the upload can be done: HTTP/1.1 200 OK ``` -If the upload cannot proceed, the server `MUST` return an appropriate HTTP status code and a custom header `X-Upload-Message` with a human readable error message. +If the upload cannot proceed, the server `MUST` return an appropriate `4xx` HTTP status code and a custom header `X-Upload-Message` with a human readable error message. + Some examples of error messages: @@ -44,6 +51,21 @@ HTTP/1.1 400 Bad Request X-Upload-Message: Invalid Content-Digest header format. Expected format: sha-256=:: ``` +```http +HTTP/1.1 401 Unauthorized +X-Upload-Message: Authorization required for uploading video files +``` + +```http +HTTP/1.1 401 Forbidden +X-Upload-Message: Account banned +``` + +```http +HTTP/1.1 411 Length Required +X-Upload-Message: Missing Content-Length or X-Content-Length headers +``` + ```http HTTP/1.1 413 Content Too Large X-Upload-Message: File too large. Max allowed size is 100MB From 8e633d00b9394dc200bac3ec2b30ad4ab5090af7 Mon Sep 17 00:00:00 2001 From: quentintaranpino Date: Thu, 12 Sep 2024 09:44:20 +0200 Subject: [PATCH 09/10] Change X-Content-Digest for X-SHA-256 --- buds/06.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/buds/06.md b/buds/06.md index d302095..0584116 100644 --- a/buds/06.md +++ b/buds/06.md @@ -10,11 +10,11 @@ Defines how clients can verify if the upload can be completed before sending the ## HEAD /upload - Upload requirements -The `HEAD /upload` endpoint `MUST` use the `Content-Digest`, `X-Content-Type` and `X-Content-Length` headers sent by client to get the SHA-256 hash, MIME type and size of the blob that will be uploaded, returning a HTTP status code and a custom header `X-Upload-Message` to indicate some human readable message about the upload requirements. +The `HEAD /upload` endpoint `MUST` use the `X-SHA-256`, `X-Content-Type` and `X-Content-Length` headers sent by client to get the SHA-256 hash, MIME type and size of the blob that will be uploaded, returning a HTTP status code and a custom header `X-Upload-Message` to indicate some human readable message about the upload requirements. ### Headers -- `Content-Digest`: A string formatted as `sha-256=::`, where `` is the base64-encoded SHA-256 hash of the blob. +- `X-SHA-256`: A string that represents the blob's SHA-256 hash. - `X-Content-Length`: An integer that represents the blob size in bytes. - `X-Content-Type`: A string that specifies the fblobile's MIME type, like `application/pdf` or `image/png`. - `X-Upload-Message`: A human readable message that explains the reason why the upload cannot proceed. @@ -32,7 +32,7 @@ Example request from the client: ```http X-Content-Type: application/pdf X-Content-Length: 184292 -Content-Digest: SHA-256=:88a74d0b866c8ba79251a11fe5ac807839226870e77355f02eaf68b156522576: +X-SHA-256: 88a74d0b866c8ba79251a11fe5ac807839226870e77355f02eaf68b156522576 ``` Example response from the server if the upload can be done: @@ -48,27 +48,27 @@ Some examples of error messages: ```http HTTP/1.1 400 Bad Request -X-Upload-Message: Invalid Content-Digest header format. Expected format: sha-256=:: +X-Upload-Message: Invalid X-SHA-256 header format. Expected a string. ``` ```http HTTP/1.1 401 Unauthorized -X-Upload-Message: Authorization required for uploading video files +X-Upload-Message: Authorization required for uploading video files. ``` ```http HTTP/1.1 401 Forbidden -X-Upload-Message: Account banned +X-Upload-Message: SHA-256 hash banned. ``` ```http HTTP/1.1 411 Length Required -X-Upload-Message: Missing Content-Length or X-Content-Length headers +X-Upload-Message: Missing X-Content-Length header. ``` ```http HTTP/1.1 413 Content Too Large -X-Upload-Message: File too large. Max allowed size is 100MB +X-Upload-Message: File too large. Max allowed size is 100MB. ``` ```http From 53472888d30cbb46f234d8fa8e429a54c7c02fdc Mon Sep 17 00:00:00 2001 From: quentintaranpino Date: Mon, 16 Sep 2024 09:21:38 +0200 Subject: [PATCH 10/10] fix forbidden response example status code --- buds/06.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/buds/06.md b/buds/06.md index 0584116..9ffb571 100644 --- a/buds/06.md +++ b/buds/06.md @@ -57,7 +57,7 @@ X-Upload-Message: Authorization required for uploading video files. ``` ```http -HTTP/1.1 401 Forbidden +HTTP/1.1 403 Forbidden X-Upload-Message: SHA-256 hash banned. ```