API calls from shell oddly slow

ttavoy

*** sorry, completely failed to figure out how to post code here ... this post might not make much sense as is, sorry about that ***

Why does this program run 6 times more slowly in an Opalstack shell than it does in a free-tier GCP (google cloud platform) shell?

The program is so simple that there doesn't seem to be much scope for me to be doing something wrong, but it is always possible.

For each of the 149 email addresses in my account it prints a line of text giving the email address, the list of destinations, and the list of forwards.

On GCP it takes 80 secs, and in Opalstack it takes 520 secs.

As a point of reference, running it locally on Windows (with each API call going separately through a low-earth-orbit starlink connection) takes only a bit longer than Opalstack, at 615 secs.

#!/usr/bin/env python3
from requests.auth import AuthBase
import requests
import json

mytoken = "XXXXXX" # replace with your token
class TokenAuth(AuthBase):
    def __init__(self, token):
        self.token = token

    def __call__(self, r):
        r.headers['Authorization'] = f"Token {self.token}"
        r.headers['Content-Type'] = "application/json"
        return r

def jpiMailList ():
  r = requests.get(f'https://my.opalstack.com/api/v1/address/list/',
                   auth=TokenAuth(mytoken))
  return r.json()

def jpiMailRead (auid):
  r = requests.get(f"https://my.opalstack.com/api/v1/address/read/{auid}",
                   auth=TokenAuth(mytoken))
  return r.json()

def jpiMailuserRead (a):
  r = requests.get(f"https://my.opalstack.com/api/v1/mailuser/read/{a}",
                   auth=TokenAuth(mytoken))
  return r.json()

def destination2name (u):
   return jpiMailuserRead (u)['name']

for i in jpiMailList ():
   e = jpiMailRead (i['id'])
   destinations = list (map (destination2name, e['destinations']))
   print (f"{e['source']} : {', '.join (sorted (destinations))} ; {', '.join (sorted (e['forwards']))}")

sean

ttavoy sorry, completely failed to figure out how to post code here

I fixed the formatting for you.

To post a code block, put it between two lines of triple-backticks (with an optional language identifier on the first line), like so:

```python
print('hello world')
```

The result:

print('hello world')

sean

This runs pretty slowly on my own system as well, completed in 311 secs. I'm going to guess that GCP has better network latency and that it's slower on your server because the server is located on a different continent than the API server.

It's slow because of the high number of API calls, ie:

1 call to address/list
1 call to address/read for each result in address/list
1 call to maibox/read for each destination mailbox on each result in address/list

If you have a lot of addresses then this is potentially hundreds or thousands of API calls.

The code could be optimized quite a bit:

address/list returns a list of the same objects you get from address/read so jpiMailRead isn't needed.
You can use the embed parameter (documented here) to include the mailbox information in the result of address/list, eliminating the need for jpiMailuserRead.

The resulting code is a single API call and is quite fast:

#!/usr/bin/env python3
from requests.auth import AuthBase
import requests

mytoken = "XXXXXX" # replace with your token
class TokenAuth(AuthBase):
    def __init__(self, token):
        self.token = token

    def __call__(self, r):
        r.headers['Authorization'] = f"Token {self.token}"
        r.headers['Content-Type'] = "application/json"
        return r

def jpiMailList ():
  r = requests.get(f'https://my.opalstack.com/api/v1/address/list/?embed=destinations',
                   auth=TokenAuth(mytoken))
  return r.json()

for i in jpiMailList ():
   print (f"{i['source']} : {', '.join (sorted(x['name'] for x in i['destinations']))} ; {', '.join (sorted (i['forwards']))}")

This runs in under 2 seconds on my system.

And here's a 1-line version just for fun 🙂

import requests; print('\n'.join([f"{i['source']} : {', '.join (sorted(x['name'] for x in i['destinations']))} ; {', '.join (sorted (i['forwards']))}" for i in requests.get(f'https://my.opalstack.com/api/v1/address/list/?embed=destinations',headers={'Authorization':'Token XXXXXX','Content-Type':'application/json'}).json()]))

ttavoy

That is amazing and very educative ... thank you!