Hello, Buckets

Stedi Buckets is worth getting excited about, or at least so I’m told. We launched it together with Stedi SFTP and I can’t help but feel that SFTP stole the show that day. Receive documents from customers without hassle? That is solving a business problem. But storing those documents? Sure, it’s necessary, but there’s not a lot for me to do with documents that just sit there. Unless you give me programmatic access. Then I can do with them whatever I want and that gets my coder’s heart pumping.

Setting a challenge

Stedi SFTP and Stedi Buckets are related and I can access both of them using the SDK. I need a starting point though, so I’ll set myself a challenge. I want to generate a report that tells me how many files each of my customers has sent me. I also want to know how many of those files contain EDI. What I’m looking for is something like this.

user: Paper Wrap - files: 10 - EDI: 8
user: Blank Billing - files: 4 - EDI: 2
user: Blank Shipping - files: 11 - EDI: 7
user: Blank Management - files: 17 - EDI: 9

Let me stop pretending that I’m making this up as I go: I already finished the challenge. It turned out easier than I thought. There, I spoiled the ending, so you might as well stop reading. Actually, that’s not a bad idea. If you feel you can code this yourself, then open the Stedi SFTP SDK reference and the Stedi Buckets SDK reference and go for it.

• If you want a little more help, continue reading Programming with the SDK.

• If you want a lot more help, jump to Walkthrough.

Programming with the SDK

Assuming that installing Node.js is something you’ve done already, you can get started with the SDK by running the following commands in your project directory.

npm install @stedi/sdk-client-buckets
npm install @stedi/sdk-client-sftp

Since I’ve already finished the challenge, I know exactly which operations I’m going to need.

• List all SFTP users.

• List all objects in a bucket.

• Download the contents of an object.

List all SFTP users

const sftp = require("@stedi/sdk-client-sftp");

async function main() {
  const sftpClient = new sftp.Sftp({
    region: "us",
    apiKey: process.env.STEDI_API_KEY,
  });

  const listUsersResult = await sftpClient.listUsers({});
  const users = listUsersResult.items;
  console.info(users);
}

main();

List all objects in a bucket

const buckets = require("@stedi/sdk-client-buckets");

async function main() {
  const bucketsClient = new buckets.Buckets({
    region: "us",
    apiKey: process.env.STEDI_API_KEY,
  });

  const listObjectsResult = await bucketsClient.listObjects({
    bucketName: "YOUR BUCKET NAME HERE",
  });
  const objects = listObjectsResult.items || []; // items is undefined if the bucket doesn’t contain objects
  console.info(objects);
}

main();

Download the contents of an object

const buckets = require("@stedi/sdk-client-buckets");
const consumers = require("stream/consumers");

async function main() {
  const bucketsClient = new buckets.Buckets({
    region: "us",
    apiKey: process.env.STEDI_API_KEY,
  });

  const getObjectResult = await bucketsClient.getObject({
    bucketName: "YOUR BUCKET NAME HERE",
    key: "YOUR OBJECT KEY HERE",
  });
  const contents = await consumers.text(getObjectResult.body);
  console.info(contents);
}

main();

Paging

The list-operations only return the first page of results. Let’s not settle for that.

const buckets = require("@stedi/sdk-client-buckets");

async function main() {
  const bucketsClient = new buckets.Buckets({
    region: "us",
    apiKey: process.env.STEDI_API_KEY,
  });

  let objects = [];
  let pageToken = undefined;
  do {
    const listObjectsResult = await bucketsClient.listObjects({
      bucketName: "YOUR BUCKET NAME HERE",
      pageSize: 5,
      pageToken: pageToken,
    });
    objects = objects.concat(listObjectsResult.items || []);

    pageToken = listObjectsResult.nextPageToken;
  } while (pageToken !== undefined);

  console.info(objects);
}

main();

Walkthrough

The devil is in the details. The code above shows you what you need to program with the SDK, but it doesn’t explain anything. Allow me to rectify.

From time to time, I’ll do something on the command line. To follow along, you need to run a Linux shell, the Mac terminal, or Windows Subsystem for Linux.

Installing Node.js

1. Install Node.js.

The Stedi SDK is written for JavaScript. It also works with languages that compile to JavaScript—like TypeScript, CoffeeScript, and Haxe—but it requires a JavaScript environment. In practice, that means Node.js.

How you get up and running with Node.js depends on your operating system. I’m not going to cover all the options here. Fortunately, the official website describes how to install Node.js.

Creating a project

2. Create a folder for your project.

mkdir stedi-challenge
cd stedi-challenge

3. Create a file called main.js.

touch main.js

4. Open the file in your preferred code editor.

5. Paste the following code into the file.

console.info("Hello, world!");

6. Run the code.

node main.js

This should output the following:

Hello, world

Creating a Stedi-account

7. Go to the sign-up page and create a free Stedi-account.

If you already have Stedi-account, you can use that one. If you’re already using Stedi SFTP, then your results might look a little different, but it will all still work.

Create an API-key

8. Sign in to the Stedi Dashboard.

9. Open the menu on the top left.

10. Under Account, click API Keys.

11. Click on Generate API key.

12. Enter a description for the API key. If you have multiple API keys in your account, you can use the description to tell them apart. Other than that, it doesn’t matter what you fill in. I usually just type my name.

13. Click on Generate.

14. Copy the API key and store it somewhere safe.

15. Click Close.

You need the API key to work with the SDK. It tells the SDK which account to connect to. It’s important you keep the API key safe, because anyone who has your API key can run code that access your account.

Test the Stedi Buckets SDK

16. Install the Stedi Buckets SDK.

npm install @stedi/sdk-client-buckets

17. Open main.js.

18. Remove all code.

19. Import the Stedi Buckets package.

const buckets = require("@stedi/sdk-client-buckets");

20. Create a Stedi Buckets client.

const bucketsClient = new buckets.Buckets({
  region: "us",
  apiKey: process.env.STEDI_API_KEY,
});

21. List all buckets in your account.

async function main() {
  const listBucketsResult = await bucketsClient.listBuckets({});
  const buckets = listBucketsResult.items;
  console.info(buckets);
}

main();

22. On the command line, set the environment variable STEDI_API_KEY to your API key.

export STEDI_API_KEY=YOUR.API.KEY.HERE

23. Run the code.

node main.js

This should output the following:

[]

The client allows you to send commands to Stedi Buckets. You must pass it a region, although we only support us right now. You also give it your API key, so it knows which account to connect to. You could paste your API key directly into the code, but then everyone who has access to your code, can see your API key and you should keep it safe.

Instead, this code reads the API key from an environment variable that you set on the command line. This way, the API key is only available to you. If someone else wants to run the code, they need to have their own API key and set it on their command line.

To get a list of all buckets, you call listBuckets(). The functions in the SDK expect all their parameters wrapped in an object. listBuckets() doesn’t have any required parameters, but it still expects an object. That’s why you have to pass in {}.

listBuckets() is an async function, so you have to await it, but you can only use await inside another async function. That’s why the code is wrapped inside the function main().

Since you don’t have any buckets in your account yet, the output is an empty array.

Test the Stedi SFTP SDK

24. Install the Stedi SFTP SDK.

npm install @stedi/sdk-client-sftp

25. Remove all code from main.js.

26. Import the Stedi SFTP package.

const sftp = require("@stedi/sdk-client-sftp");

27. Create a Stedi SFTP client.

const sftpClient = new sftp.Sftp({
  region: "us",
  apiKey: process.env.STEDI_API_KEY,
});

28. List all SFTP users in your account.

async function main() {
  const listUsersResult = await sftpClient.listUsers({});
  const users = listUsersResult.items;
  console.info(users);
}

main();

29. Run the code.

node main.js

This should output the following:

[]

As you can see, this works just like the Stedi Buckets SDK. The same notes apply.

Create an SFTP user

30. Open the Stedi Dashboard.

31. Open the menu on the top left.

32. Under Products, click on SFTP.

33. Click on Create User.

34. Fill in the form with the values you see below.

35. Click Save.

36. Copy the connection string and password and store it somewhere safe.

37. Click Done.

This creates a user that has access to the directory /paper-wrap on the SFTP server.

Paper Wrap is a store that sells edible gift wrapping. They care greatly about preventing paper waste, so they do all their business electronically.

The SFTP server is ready for use, even though you didn’t create a bucket for it. Stedi does this for you automatically.

Find the bucket name

398. Add the following code at the end of main().

if (users.length == 0) {
  console.info("No users.");
  return;
}

const bucketName = users[0].bucketName;
console.info(bucketName);

39. Run the code.

node main.js

This should output the following, although your bucket name and username will be different.

[
  {
    bucketName: 'e97e59c8-d5d4-4dde-97ab-52cdf777eaf5-sftp',
    description: 'Paper Wrap',
    endpoint: 'data.sftp.us.stedi.com',
    homeDirectory: '/paper-wrap',
    lastConnectedDate: undefined,
    username: 'P7EGRE9H'
  }
]
e97e59c8-d5d4-4dde-97ab-52cdf777eaf5-sftp

It’s convenient that Stedi creates the SFTP bucket for you, but you need the name of the bucket if you want to access it from code. You could get the name from the dashboard, but you didn’t get into programming to copy and paste things from the UI.

When you retrieve information about a user, it includes the bucket name in a field conveniently called bucketName. All SFTP users use the same bucket, so you can get the bucket name from the first user and use it throughout.

Create more SFTP users

40. In the Stedi Dashboard, click on Create User.

41. Fill in the form with the values you see below.

42. Click Save.

43. Copy the connection string and password and store it somewhere safe.

44. Click Done.

45. Click on Create User.

46. Fill in the form with the values you see below.

47. Click Save.

48. Copy the connection string and password and store it somewhere safe.

49. Click Done.

50. Click on Create User.

51. Fill in the form with the values you see below.

52. Click Save.

53. Copy the connection string and password and store it somewhere safe.

54. Click Done.

Blank is a manufacterer of invisible ink. They went fully digital after the court ruled that their paper contracts weren’t legally binding.

Blank has separate departments for billing and shipping and both departments get their own user with their own directory. This way, Shipping can’t accidentally put their ship notices among the invoices. The general manager likes to keep an eye on everything, so he has a user that can see documents from both Shipping and Billing, but can’t access Paper Wrap’s documents.

Create test files

55. Create a file called not_edi.txt.

56. Add the following content to not_edi.txt.

This is not an EDI file

57. Create a file called edi.txt.

58. Add the following content to edi.txt.

ISA*00*          *00*          *ZZ*PAPER WRAP     *ZZ*BLANK          *210901*1234*U*00801*000000001*0*T*>~
GS*PO*SENDERGS*007326879*20210901*1234*1*X*008020~
ST*850*000000001~
BEG*24*SP*PO-00001**20210901~
PO1**5000*04*0.0075*PE*GE*Lemon Juice Gift Card~
SE*1*000000001~
GE*1*1~
IEA*1*000000001

If the script is going to count files, you should give it some files to count. The contents don’t really matter, other than that some files should contain EDI. You’ll upload copies of these two files to the SFTP server.

A word about SFTP clients

I’m about to show you how to upload files to the SFTP server and I’m going to do it from the command line. I think that’s convenient since I’m doing a lot of things on the command line already, but it’s not the only way. If you prefer dragging and dropping your files, you can install an FTP client like FileZilla. You won’t be able to follow the instructions in the next paragraph step by step, but it shouldn’t be too hard to adapt them. Here are some pointers.

• You can find the username and host on the Stedi Dashboard.

• Host and SFTP endpoint are the same thing.

• If you need to specify a port, use 22.

• Call the files on the SFTP server whatever you like; it doesn’t matter to the code.

• The exact number of files you upload doesn’t matter; just gives every user a couple of files.

Upload test files

59. On the command line, make sure you’re in the directory that contains the files edi.txt and not-edi.txt.

60. Connect to the SFTP server with the connection string from the Paper Wrap user. Your connection string will look a little different than the one in example.

sftp 9UE5Z386@data.sftp.us.stedi.com

61. Enter the password for the Paper Wrap user.

62. Upload a couple of files.

put edi.txt coconut-christmas
put edi.txt toffee-birthday
put edi.txt blueberry-ribbon
put edi.txt cinnamon-surprise
put edi.txt salty-sixteen
put edi.txt beefy-graduation
put edi.txt cashew-coupon
put edi.txt bittersweet-valentine
put not-edi.txt whats-that-smell
put not-edi.txt dont-ship-the-fish

63. Disconnect from the SFTP server.

bye

64. Connect to the SFTP server with the connection string from the Blank Billing user.

65. Enter the password for the Blank Billing user.

66. Upload a couple of files.

put edi.txt pay-me
put edi.txt seriously-pay-me
put not-edi.txt i-know-where-you-live
put not-edi.txt thank-you

67. Disconnect from the SFTP server

68. Connect to the SFTP server with the connection string from the Blank Shipping user.

69. Enter the password for the Blank Shipping user.

70. Upload a couple of files.

put edi.txt too-heavy
put edi.txt is-this-empty
put edi.txt poorly-wrapped
put edi.txt other-side-down
put edi.txt handle-with-gloves
put edi.txt negative-shelf-space
put edi.txt destination-undisclosed
put not-edi.txt empty-labels
put not-edi.txt boxes-and-bows
put not-edi.txt be-transparent
put not-edi.txt how-to-drop-without-breaking

71. Disconnect from the SFTP server.

72. Connect to the SFTP server with the connection string from the Blank Management user.

73. Enter the password for the Blank Management user.

74. Upload a couple of files.

put not-edi.txt strategic-strategizing
put not-edi.txt eat-my-spam

List all files

75. Create a Stedi Buckets client. Add the following code at the top of main.js.

const buckets = require("@stedi/sdk-client-buckets");

const bucketsClient = new buckets.Buckets({
  region: "us",
  apiKey: process.env.STEDI_API_KEY,
});

76. Get a list of files from the SFTP server. Add the following code to the end of main().

const listObjectsResult = await bucketsClient.listObjects({
  bucketName: bucketName,
});
const objects = listObjectsResult.items || [];
console.info(objects);

77. Run the code.

node main.js

This should output the following:

[
  {
    bucketName: 'e97e59c8-d5d4-4dde-97ab-52cdf777eaf5-sftp',
    description: 'Blank Management',
    endpoint: 'data.sftp.us.stedi.com',
    homeDirectory: '/blank',
    lastConnectedDate: '2022-07-28',
    username: '8X0HLJJW'
  },
  {
    bucketName: 'e97e59c8-d5d4-4dde-97ab-52cdf777eaf5-sftp',
    description: 'Blank Shipping',
    endpoint: 'data.sftp.us.stedi.com',
    homeDirectory: '/blank/shipping',
    lastConnectedDate: '2022-07-28',
    username: '4UBJ9H9C'
  },
  {
    bucketName: 'e97e59c8-d5d4-4dde-97ab-52cdf777eaf5-sftp',
    description: 'Paper Wrap',
    endpoint: 'data.sftp.us.stedi.com',
    homeDirectory: '/paper-wrap',
    lastConnectedDate: '2022-07-28',
    username: 'P7EGRE9H'
  },
  {
    bucketName: 'e97e59c8-d5d4-4dde-97ab-52cdf777eaf5-sftp',
    description: 'Blank Billing',
    endpoint: 'data.sftp.us.stedi.com',
    homeDirectory: '/blank/billing',
    lastConnectedDate: '2022-07-28',
    username: '23SO1YKM'
  }
]
e97e59c8-d5d4-4dde-97ab-52cdf777eaf5-sftp
[
  {
    key: 'blank/billing/i-know-where-you-live',
    updatedAt: '2022-07-28T15:21:23.000Z',
    size: 24
  },
  {
    key: 'blank/billing/pay-me',
    updatedAt: '2022-07-28T15:21:22.000Z',
    size: 295
  },
  {
    key: 'blank/billing/seriously-pay-me',
    updatedAt: '2022-07-28T15:21:22.000Z',
    size: 295
  },
  {
    key: 'blank/billing/thank-you',
    updatedAt: '2022-07-28T15:21:24.000Z',
    size: 24
  },
  {
    key: 'blank/eat-my-spam',
    updatedAt: '2022-07-28T15:25:22.000Z',
    size: 24
  },
  {
    key: 'blank/shipping/be-transparent',
    updatedAt: '2022-07-28T15:22:26.000Z',
    size: 24
  },
  {
    key: 'blank/shipping/boxes-and-bows',
    updatedAt: '2022-07-28T15:22:25.000Z',
    size: 24
  },
  {
    key: 'blank/shipping/destination-undisclosed',
    updatedAt: '2022-07-28T15:22:24.000Z',
    size: 295
  },
  {
    key: 'blank/shipping/empty-labels',
    updatedAt: '2022-07-28T15:22:24.000Z',
    size: 24
  },
  {
    key: 'blank/shipping/handle-with-gloves',
    updatedAt: '2022-07-28T15:22:23.000Z',
    size: 295
  },
  {
    key: 'blank/shipping/how-to-drop-without-breaking',
    updatedAt: '2022-07-28T15:22:26.000Z',
    size: 24
  },
  {
    key: 'blank/shipping/is-this-empty',
    updatedAt: '2022-07-28T15:22:21.000Z',
    size: 295
  },
  {
    key: 'blank/shipping/negative-shelf-space',
    updatedAt: '2022-07-28T15:22:23.000Z',
    size: 295
  },
  {
    key: 'blank/shipping/other-side-down',
    updatedAt: '2022-07-28T15:22:22.000Z',
    size: 295
  },
  {
    key: 'blank/shipping/poorly-wrapped',
    updatedAt: '2022-07-28T15:22:21.000Z',
    size: 295
  },
  {
    key: 'blank/shipping/too-heavy',
    updatedAt: '2022-07-28T15:22:20.000Z',
    size: 295
  },
  {
    key: 'blank/strategic-strategizing',
    updatedAt: '2022-07-28T15:25:21.000Z',
    size: 24
  },
  {
    key: 'paper-wrap/beefy-graduation',
    updatedAt: '2022-07-28T15:20:11.000Z',
    size: 295
  },
  {
    key: 'paper-wrap/bittersweet-valentine',
    updatedAt: '2022-07-28T15:20:12.000Z',
    size: 295
  },
  {
    key: 'paper-wrap/blueberry-ribbon',
    updatedAt: '2022-07-28T15:20:09.000Z',
    size: 295
  },
  {
    key: 'paper-wrap/cashew-coupon',
    updatedAt: '2022-07-28T15:20:12.000Z',
    size: 295
  }
]

You can’t get files per user, because Stedi Buckets doesn’t know anything about users. You also can’t get files per directory, because technically Stedi Buckets doesn’t have directories. That’s a topic for another time, though.

You can list all the files. The result from listObjects() contains an array called items with information on each file. If there are no files on the SFTP server, items will be undefined. For our code, it’s more convenient to have an empty array if there are no files, hence the expression listObjectsResult.items || [].

If you take a close look at the output—and you’ve been following this walkthrough to the letter—you’ll discover that it doesn’t contain every single file. listObjects() only return the first 25. To get the rest as well, you’ll need paging.

Paging through files

78. Page through the files on the SFTP server. Replace the code from the previous paragraph with the following.

let objects = [];
let pageToken = undefined;
do {
  const listObjectsResult = await bucketsClient.listObjects({
    bucketName: bucketName,
    pageToken: pageToken,
  });
  objects = objects.concat(listObjectsResult.items || []);

  pageToken = listObjectsResult.nextPageToken;
} while (pageToken !== undefined);

console.info(objects.length);

79. Run the code.

node main.js

This should output the following:

27

When there are more results to fetch, listObjects() will add the field nextPageToken to its result. If you then call listObjects() again, passing in the page token, you will get the next page of results. The first time you call listObjects(), you won’t have a page token yet, so you can pass in undefined to get the first page.

You can use concat() to put the files of each page into a single array.

Paging through users

80. Replace the line let pageToken = undefined; with the following.

pageToken = undefined;

81. Page through all SFTP users. Replace the code that lists users, at the beginning of main(), with the following.

let users = [];
let pageToken = undefined;
do {
  const listUsersResult = await sftpClient.listUsers({
    pageToken: pageToken,
  });
  users = users.concat(listUsersResult.items);

  pageToken = listUsersResult.nextPageToken;
} while (pageToken !== undefined);

There are only four users, so for this challenge, paging through the users won’t make a difference, but it makes the scripts more robust. It works just like paging through files.

Counting files

82. For every user, loop through all files and count the ones that are in their home directory. Add the following code to the end of main().

for (let user of users) {
  const homeDirectory = user.homeDirectory.substring(1);

  let fileCount = 0;
  for (let object of objects) {
    if (object.key.startsWith(homeDirectory)) {
      fileCount++;
    }
  }

  console.info(`user: ${user.description} - files: ${fileCount}`);
}

83. Run the code.

node main.js

This should output the following:

user: Blank Management - files: 17
user: Blank Shipping - files: 11
user: Paper Wrap - files: 10
user: Blank Billing - files: 4

Every file has a field called key, which contains the full path, for example blank/billing/thank-you. If the start of the key is the same as the user’s home directory, then the user owns the file. The only problem is that the home directory has a / at the start and the key doesn’t. user.homeDirectory.substring(1) strips the / from the home directory.

Detecting EDI

84. Import the stream consumer package. Add the following code at the top of main.js.

const consumers = require("stream/consumers");

85. Download each file from the SFTP server. Add the following code right below fileCount++.

const getObjectResult = await bucketsClient.getObject({
  bucketName: bucketName,
  key: object.key,
});
const contents = await consumers.text(getObjectResult.body);

86. Add a counter for EDI files. Add the following code right below let fileCount = 0.

let ediCount = 0;

87. Determine if the file contents is EDI. Add the following right below the previous code.

if (contents.startsWith("ISA")) {
  ediCount++;
}

88. Output the result. Replace the last line that calls console.info() with the following code.

console.info(`user: ${user.description} - files: ${fileCount} - EDI: ${ediCount}`);

This should output the following:

user: Blank Management - files: 17 - EDI: 9
user: Blank Shipping - files: 11 - EDI: 7
user: Paper Wrap - files: 10 - EDI: 8
user: Blank Billing - files: 4 - EDI: 2

When you call getObject() the result includes a stream that allows you to download the contents of the file. The easiest way to do this, is using consumer.text() from Node.js’s stream consumer package.

To detect if a document is EDI, you can check if it starts with the letters ISA. It’s easy to do and I expect it covers at least 99% of cases, so I call that good enough.

Challenge completed

const buckets = require("@stedi/sdk-client-buckets");
const sftp = require("@stedi/sdk-client-sftp");
const consumers = require("stream/consumers");

const bucketsClient = new buckets.Buckets({
  region: "us",
  apiKey: process.env.STEDI_API_KEY,
});

const sftpClient = new sftp.Sftp({
  region: "us",
  apiKey: process.env.STEDI_API_KEY,
});

async function main() {
  let users = [];
  let pageToken = undefined;
  do {
    const listUsersResult = await sftpClient.listUsers({
      pageToken: pageToken,
    });
    users = users.concat(listUsersResult.items);

    pageToken = listUsersResult.nextPageToken;
  } while (pageToken !== undefined);

  if (users.length == 0) {
    console.info("No users.");
    return;
  }

  const bucketName = users[0].bucketName;
  console.info(bucketName);

  let objects = [];
  pageToken = undefined;
  do {
    const listObjectsResult = await bucketsClient.listObjects({
      bucketName: bucketName,
      pageToken: pageToken,
    });
    objects = objects.concat(listObjectsResult.items || []);

    pageToken = listObjectsResult.nextPageToken;
  } while (pageToken !== undefined);

  console.info(objects.length);

  for (let user of users) {
    const homeDirectory = user.homeDirectory.substring(1);

    let fileCount = 0;
    let ediCount = 0;
    for (let object of objects) {
      if (object.key.startsWith(homeDirectory)) {
        fileCount++;

        const getObjectResult = await bucketsClient.getObject({
          bucketName: bucketName,
          key: object.key,
        });
        const contents = await consumers.text(getObjectResult.body);
        if (contents.startsWith("ISA")) {
          ediCount++;
        }
      }
    }

    console.info(`user: ${user.description} - files: ${fileCount} - EDI: ${ediCount}`);
  }
}

main();