API Authentification & Authorization 3 part Tutorial

The first branch Auth was originally a javascript that I downloaded and simplified (I think from WebDev Simplified). I then converted it to Typescript since that is my first choice for developing. I removed all database and any front end coding/cookies. All of this to give someone a really easy to read and follow route and controllers in on. Limiting too many concepts at once.

Scriptsnpm run dev, 2) npm run test, 3) view results by opening tests.log file.

When you want to modify the API server code, I strongly recommend vs code. You will want to launch muliple terminalsThe 1st one wil run the script npm run watch this keeps Typescript recompiling whenever a *.ts file is saved. You will have to restart the server. ( I reallize I could have implement ts-node and nodemon, but just more complexity for the average joe. I

I also provide 2 launch configuration scripts for vs code. One is for the dev server. Process is watch, then f5 for debug, the run test script. I usually comment out tests and add new tests as I want for the moment. This way it is shorter run. I could have used Postman, thunderclient, but at the time of this creattion there were a few accessibility issues and I was forced to create my own. The test script maintains an in memory list of emails associated with a web token. I treated emails as userids, since I do no validation in this demo/tutorial.

I like to place server code in editor group 1 (ctrl 1). Test.js script in editor group 2 (ctrl+2), Test results tests.log file in editor group 3 (ctrl 3). I press ctrl ` for the terminal and run the test (npm run test). Check results (ctrl+3). Review results highlight all results and edelete and save. then rerun. It is nice it automatically detects that the files have changed and loads the changes. ** warning if you deleted and forgot to save it wil not refresh and be out of sync for the log file. You will then have to close log file w/o saving, and reopen it.

Copy the code below for cloning andplace in a folder where you want git to create the cloned folder.

It may not automatically download all branches. You test by trying to switch to branch. git checkout Auth If you get an error then you will have to fetch the other branches with the command below. git fetch --all Now you should not get any errors switching to the branch. Finally check status. git status This will always tell you what branch you are on. The main branch is a good match for the final branch "mvc".

This will require you to click the 3 download links below. Unzip to a folder. Reccoment and sub folder for each downloaded folder.

Zip File: 1) Auth branch. 2)Role branch 3) MVC branch

You will need a web token to run in each branch

So you will have to create one in each branch or copy it into each branch individually. Now I went ahead and allowed my web token I used in this demo to be uploaded to github. This is a no no, but if you are developing and testing locally it should not be an issue. Use my web token provided at your own risk, since anyone who downloads this API server will have it too. It is easy to create so I recommend that you do create one..

The above code will create a web token and store in a .token file. Simply copy the entire token and replace the token I provide in the .env file. That is it. Now repeat in every branch.

***Note: Git users you can easily switch between branches with the following command: git chickout role Replace the branch name role I used with auth or mvc for those branches You can verify where you are by : git status . If you get a messesage that files have been modified and will be lost. Simply commit them or just do a git restore . to abandon all changes, then you will be able to switch.

Installation of dependencies

You simply need to do npm install. This command will install all you will need for that branch. You may have to do this for every branch as well. Since more packages were added after Auth branch & Role Branch.

Congratulations you are now ready to start learning how to do authentification using web Tokens and Express Server.

Just run the server and hit it with API tests To tun the server without debugging simply type: npm run dev. The server wil be launched on port 3000 as the default. You can change the port in .env, but my Testscripts depend on this port. You can use Postman, ThunderClient, or curl or any other api tester. I provided a test.js script file to run against this API server. I put this API through the wringer.

To run my custom test.js file simply type: npm run test. This will run through a whole lot of tests. You can comment out test you do not want to do, or modify any of them as you desire. Te test results will be displayed in the terminal, but there are so many they will scroll off. So I also create a test.log file. Simply pull this file up and review the results. I declare the test and display the result from the API Server. Now in this test and API Server I abuse the email variable. Email is a good user identifier, because it is unique. However, it does not lend itself in blogs or role end points. Normally you create a username field for this purpose. Too complicated and I want this to be simple, so I just use a user first name as email.

The users I used are some family members and myself. The passwords I used are made up, except "letmein007" (a password I used in supporting clients back in 1996 through 2007 for remote connections. That connection software and VAR I worked for are non existent today) Besides today's standards it is a weak password. This API performs no snitizing or validation That is covered in many other tutorials out there and this is only a demo project.

Typical Development Cycle

You wil will want to have Vs Code installed , if you do not already have it. You can download at Vs Code download page.

Launch Vs Code and I typically do the following:

• Open file explorer with (ctrl+shift+e) and open src/app.ts file in editor 1
• I then switch to editor group 2. Then open test/test.js. The API server are *.ts files written Typescript and the test is written Javascript.
• Now do (ctrl+`) to open or go to current terminal. Now enter npm run watch. This will watch your typescript files for any changes and recompile them. You will of course have to restart the server to get the changes.
• Press (ctrl+shift+d) to go to debug section. Press left arrow and make sure the dev launch script is selected. This will launch the API server in development mode and you can place breakpoints and run code in the java console. Now you can press (f5) to launch the debugger. Your API server is up and running.
• Now open a new terminal (ctrl+shift+`). Type in: npm run test. You will see a whole lot of test results go by or hear them.
• Now go to editor group 3 (ctrl 3) and open the file test.log in the root folder. These are the results of the test.js file you ran.
• Edit or modify a test then run the test again, over and over. Remember results are in editor group 3, so delete all contents and save. rerun and get new results witout reloading the file. Pretty handy.
• Place a breakpoint in editor group 1, and feel free to open other files and see how it runs.

*** Note file locations were as they were in auth and role, mvc and main are under api folder. ** note You can run curl from the terminal as well. curl http://localhost:3000/ or your browser. Although these are limited to on get method type requests and no token.

This code was originally pure Javascript. I converted to Typescript my language of choice. I also simplified even more and converted it into an in -memory data source. This means when server goes up or down it starts over. That is why I have a test script file that runs a buch of tests and sets up a path for followup tests. Also PostMan, ThunderClient, HopScotch could use some accessiblity improvements at this time. For now my test.js file will have to suffice.

File Overviews

src/index.ts & src/app.ts You should leave the port number to 3000 or you will break my test.js script file. You notice that it loads the app.ts which is where the server actually starts working.

The app.ts is loaded with middleware and routes that will proccess the requests as they come in. Typescript gets converted to javascript and run by NodeJs. Once concept you need to know here, is the routes and middlewares attached are all in memory as long as the Server is listening (in case Port 3000).

The app.ts file loads and creates a decision/like tre or linked lists (even though js has no pointers). All the server does is proccess requests as they come in. Also immediately when the request a response object is created as well. These 2 objects are passed down the decision tree with app.use, app.get, and others. All stored in this tree/path. So you start from the top and go down.

Just following the export app line, you will encounter app.use( ... urlencoded) and app.use( ... json). The urlencoded is not really needed, since I do not proccess any parameters on the url line. This middleware analyze the request url and parses the parameters & more into the request parameters properties. While the json middleware will modify the request body and put it in a json format, it is currently stringified json prior to this call. Json currently is set for a 50mbs limit, this means 50 megabytes of data. Now that is extremely large for this API server, a number of 2mbs would have probably worked fine. Too large will take up resources in the server and limit how many requests you can hadle at the same time. Keep in mind objects in json are passed reference, this means any modifications will be passed on.

Middleware and routes are proccessed in order until a response is sent back, usering return response.send or response.json. Then the request has been handled so not more traveling the decision tree/path. Just exit this proccess and listen for next request. Now to keep this inline the express passes a 3rd parameter the next Function. You can call this with no parameters and middleware will simply continue down the decision tree/path. In the mvc branch we will go in details about next()

The app.post("/register" is the firts route.. You will noteice it has the "/register" as the route and it handles post requests on that route. The controller is written directly on the route, this is for your convenience and understanding. Look below and you will see code that will grab data from the request body (email, password). Since we called the json parser earlier in the middle ware the request object has it neatly avaiable in the request object. We validate the email is not already registered and verify they passed a password. We return the appropiate response for invalid data or missing data. This is were you would normally validate the input /sanitize it. If all is okay we encrypt password, save to database , & generate a token and use our stored token in the .env file to generate the the token.

The token mainly 3 parts, payload, token signature/token key, expiration.key, Payload in this case is user object returned when we created the user. You want to keep the payload small, but hold some common information that you well need quite often. This will reduce round trips to the database file. The token key comes from the .env file that containes the token you stored there during setup. The expiration is when you want the token generated to expire. For this lesson I set it in the .env to 30 seconds. This way I can test the expiration function in plenty of time. Most time you would set it to a short time of 15 minutes or 2 hours. Since clients use apps to interact with your API it is genrally okay and not that annoying to the client to have the app in the background re-login. If you want a truly more secure and more convenient you can create refresh tokens as well. See my resources where you can find out how. A more advanced implementaton.

The next route is the post."/login" route and it works quite similar to the "/register" route. The next route is the first app.get("/"...) route that displays all the available routes in the auth branch. This route and all prior are unrestricted. The next route is restricted.

The app.get("/welcome", auth , ...) route is the beginning of the restricted route. You will notice this route is not like the others, it has a route "/welcome", auth, ...) Aroute, middleware, controller code. The authenticatification middleware will short circuit if not a valid token passed. It simply returns a response.sen with appropiate error codes. I did not pay close attention to erro code numbers in the auth & role branches. The final branch I standardized my responses to be more in line. The controller code that follows the auth middleware displays a welcome message and accesses the request body to get the user. This was filled from the token payload that the auth middleware parsed and placed on the request body.

The last acutal get route is "/userslist", auth, ...). I provide this route for 2 reasons. To verify users in the system, and a second route to demonstrate how express traverses the decision tree/path. Notice this one containes an auth as well. Only the auth that folows the route gets executed and not both. You can verify this by simply removing the auth, on this route and you will have full access to this route. Since this is a get route you can use your brouser or curl. curl http://localhost:3000:/userslist While the server is running.

The last route is an app.use("*") the * is a wildcard and a catch all route. So give appropeate response.

Auth Middleware

src/middleware/auth.ts is where we check the request object and validate the token. For this API example server I only use header to pass in the token and expect to be there, although you could pass in the token in a variety methods. My stes.js uses headers only and the "x-access-token" header. There are body, query, and other ways. It is your server so you make the rules. Next I take the token in the request and decrypt using the token key from my .env that I used to encryppt it. Once token is validated I attach the token with payload on the request body. anc call next().

srce/model/users.ts Here I just create users in memory using a simple object. The object arraay serves as a key on email with a value of user object. It is here to represent a table in a database.

Now you can run and test it. See the test section for that details for it. The test/test.js file will grow in each branch as I add more features. A good testing system is absolutely needed in order to develop a robust API Server. Without it your clients will not like it and will not be back.

Overview of test/Test.js This test.js script file holds in memory tokens returned by the API Server and attaches the user/email to it. This way you can simply run once of the test functions and it will grab the token associated with the user/email. If no user/email is not given, then it will use the last token used. If an invalid user/email (not used before) then it assumes you want to run the test without a token. You will see me doing this by calling it with a user/email of "fake". A battery/collection of tests are placed in functions to run a perticular set of tests. Tests are logged in test.log file for your review. It appends to this file, so delete it or clear it out to have a fresh log.

Functions testLogin & testRegister This 2 functions build post requests with email and password as parameters. It will log the eror if one is returned or token saved if one is returned. The token is added to the buffer in an indexed object with user/email as key and value is the token. No token is sent on either of these post requests.

testRoute & testRoutePost One is a get request on a route specified. The latter is of course a post request on a particular route, this function will be pressent on the Role Branch and mvc Branch. The testRoute that builds a get accepts a uner/email parameter. It performs a lookup for the user and uses the associated token if available or present. Now the testRoutePost will add a parameter for values or data to be sent. Role and Blog in other branches will neefd this data. "/Blog" is a simple text string needed. The "/role" will need and array of roles you want to assign. Both blog and role will requre a sub path of user. For example "/role/lewis" will deal lewis's roles. Logs the result of the request. Keep in mind blog and role routes are not available in the auth branch, thus no tests present either.

Collections/Sets Test Functions

Groupins of test designed to test aspects of the API. They are functions in the test.js script file. Below is the list of them.

• registerLoginTests primary focus of these tests is for logins/registrations, tokens and saving and retrieving of tokens. Primarily tests the test.js script itself. It hits against "/welcome" route which returns a welcome message to the user. So if tokens were off a wrong welcome message would come back.
• tokenAuthTests this test test authenticated routes "/welcom" & "/userslist". You can easily rerun the tests after removeing the "auth," from the "/userslist" route. Then you will see the difference. This test is primarily for the auth branch and runs slightly differently under role & mvc branch, since I add roles, owner and guest capabilities.
• tokenExpiresTest You will need to ensure that you have set the expiration time of I say 30 seconds in the .env file.. It will register and then login (since register only returns a token if registering. Since I cannot be sure if you ran any other test set I just register & login to make sure.) Also by get a new token expiration is 30 seconds that time.
• testRoles This test set I test the /role"/<user>
• testSetRoles This function test set is in role & mvc branch only. This one I do detail stress testing of setting roles, test and see if role assigned allows the user to excercise that role's power or subtracting the role breaks their priviledges. Demonstrating, that a nrelogin is required to get the new priviledges or loose them. This wuld take a token server to invalidate a token instantly or add new priveledges. Topic for another time.

The last function is "doTests()". Now this function has a simple list of group test one after another. simply comment out the tests you do not want to run or add your own and just return from it. Last line of the test/test.js file is doTests(); that is is the actual 1st line of the script that starts the test run. Keep in mind all tests are async, so you must proceed with await keyword. Otherwise you will get irregular results.

Running the Test

To run the tests simply type: npm run test. Now the server must be running as well. You can run in debug if you want to edit it. Two launch scripts for VS Code have been provided, simply sect the launch script you desire. If you plan on editing, you will need to run npm run watch in another terminal. This will keep recompiling the typescript code into javascript in the background. You can manually recompile with the "tsc" command in the terminal as well. After an edit and save, simply restart the server to get the lates version with your modifications.

If you do not plan on debugging simply type npm run dev this will launch the API server. Then you can simply run it against that server. npm run test. You will need two terminal windows, but you do ot need vs code. You can simply launch another command prompt/linux session and do it there.

All results from thest will appear in the test.log file. Simply read through them and verify the est you want.

src/app.ts you will note at the top I import the verifyToken as auth as we did in the auth branch. Except this on comes from a custom version I created called src/middleware/role--auth.ts. Works like the previous, but creates a custom role of guest if unknown and places it on the request object. This where lw_request a extended version of request object. Simply press f12 to go to the custom definition for lw_request I created. Also added 2 more middleware functions/imports hasRoles and validRoute. Each of them will user role and owner status to verify if they have access. The auth is not on a particular route, it is prior to routes where they will need access to role & user information and it is on a app.use( auth ) line. The app.get("/welcome", hasRoles(["registered"])) Now middleware tests if the user has the specified role in order to run the controller code which is the welcom message.

Next route is app.get("/userslist", hasRoles(["guest"]), ....) now I am testing that they are note registered basically or an admin. This one is crazy, but it illustrates this functionallity perfectly.

I created 2 routes blog and role and I implement them here in src/app.ts. Normally you place the routes in a seperate file. I do it so you can see it up close and personal, without jumping around at this time.

The app.use("/blog", blogRoute...) loads the glogRoute at the routing of "/blog". BlogRoute.use( calls validUser, hasRoles...) The validUser checks the request and identifies if the current user is the owner or not. So now we know who the owner is on the request object. HasRoles grants access only to registered and editors. The next line I immediately do a blogRoute.use("/*", ) routes start from where they are attached to. This one is /blog route on the app.use. This way you can use same route code in other routes tree locations.

You will notice I daisy chane a .get off the prior route("/*"..). I gostraight to the controller code since the filtering and authorization has already been done in prior lines.

I then do a .post(hasRoles([]"editor"], true")). This route added the the Owner flag has access to this route + method (post). So you can edit only your own or you are the editor. Admins always have rights. )

The roleRoute I do pretty much the same thing. Only difference is that I deny the owner the access to the post route, thus they cannot modify their own roles. Also role must be "Lunch lady" to change roles.

src/midleware Files listing

• auth.ts This is the original auth.ts. It is here for your convenience of comparing the changes.
• authorize.ts Simply checks the request object for the property isAuthorized and returns eror response if not or goes on to next if authorized. Simplies test
• role-auth.ts There are some minor differences from the original. I only check the header for the token. Also if they do not include a token I login them as guest. If the token is invalid I deny them access.
• roles.ts This file holds the middleware for the hasRoles function. The coding here takes advantage of how javascript thinks of functions. Functions are passed as values u they have a () after them to invoke them. So what this one does is it is a function with parameters, thus making them available to the code within and returned. Notice this function returns an anonymous function. So when this function is placed on a user or route you put (parameters...). So the app.use or .get, ...etc only see the returned function which is anonymous. The anonymous function uses the parameters passed into the hasRoles function and decides if access is granted by going to next or blocks them and returns an error message.
• validUserRoute This simply validates that the route has a valid user as part of the route. Afer user fake does not exist as a user so ther is no /blog/fake route.

You will also note some error ignoring where some crazy error kept coming up, but was no affect on the results. Figure I would track it down in the mvc when I add error handling, logging and arrange in a more easily to digest manner. I am blind and find smaller code files and more focused functions to be easier to track and comprehend. For my visually impaired you will also appreciate the difference.

A whole lot has happened going from the role branch to this mvc branch. You will note a branch new src/API folder , which is used to store API routes. A large API with multi versions of the API May add versions then the actual routes. Example: src/API/v1.0 and src/API/v2.1, ... etc. There the routes will be with their specific code and midddleware calls. Now this one has no versions, but you will notice that there is an auth sub-folder. This holds the authentic validations that you may implement. This can O-Auth (Facebook, GitHub, Google, etc), externa token issuer, Appi calls to your own Token Server. Many developers out their seem to be settling on the auth folder as a sub folder on the API or routes folder.

The src/controllers is a folder where you place all controllers. The prior branches had the code for the controllers attached to the route for your convenience, this is not what you desire. Smaller code, specialized, and seprated make easier to maintain and manage code. This code acts as a communication layer between data server/api server, and presentation. For example for an API: controller may call for data from the data server and then pass it on to data manipulator to present this data in a speciallized manner or format (view). Then the controller would return the value. Controller should not do any data manipulation, the login and register clearly violate this rule, most likely should be a tonken.ts component under the auth sub-folder and there some other handling could be implemented as well (buffering). Larger applications may even do versioning similar as I mentioned in API as well. This aids in transitioning to the next version and the ability to rollback quickly if something is off.

Logging The ability for the API server to log all kinds of data/requests has been created. I created src/config/logger.ts, since I would most likely store logging information in a database when present. Although this logger only outputs the logs to the file system. You can do both and customizing it as well. I use a screen reader and this is only demo/test server I moved timestamp to the end of the log entry and shortened it. Refer to the logger.ts and simply uncomment the line above my timestamp and comment out mine. Most admins/developers use log analyzers such as Data Dog. This is Winston logging package from npm ( a very popular logger). This logger curently has 2 transports and log types defined. Notice in production mode only warn and server errors are logged, the logger ignores others. See the .env file for that setting.

Morgan is located in the src/middleware/morgan.ts is a request logger system. You can use this to track different types of requests. Want to see what requests get the highest amount of traffic? How many invalid request are bogging down your application? If your application gets DNOS (denial of servervice) attacks it will shut down your service or block all others. Morgan can help you track down these items and more. Perhaps help you create better error handling rules to block or streamline these requests. For example: This API only handles "GET" and "POST" request types. Good code wuld place a middleware filter at the top, for all invalid requests to either log & respond, respond, or ignore. By tracking some problems you can track bad IP actors out their and block them completely. It is your server and your rules. I only implemented their basic morgan logging template. You will have to refer to Morgan for more options. See resources.

Error Handling Error handling is quite versatile and simple to emplement. Special error logging meddlewareand error responders that send the response object to the client. I do a simple somewhat error handling using custom error extended classes (AppError, RoleError, TestError, ...etc..). The logging I only log it to the appropiate logger and not much specific to the route. Fore esample: RoleError could have had its own errorlog middleware and output information particular to tha route and indicate on the inherited class AppError that it was already logged. Then the Standard errorLogger would ingnore it.

Express has a a pretty neat way of handling errors. You simply pass the error object in the next(error);. Express will pass this error log down the decision tree until it finds the next middleware that accepts error object as the first parameter then request, respoinse, nextFunction. The code for the error handlers middelware I use are in src/middleware/error-handlers.ts. You will find them as middleware functions in src/app.ts in app.use( ..).

.env file contains following logging enviromentals that were added to do with logging. "log_exc_error_names" is a list of names to exclude, ie RoleError, BlogError // would exclude user feedback errors from blog and Role that throw a RoleError or BlogError. This is nice when you want to focus on a particular group of errors that users are getting. "log_inc_error_codes" this one is the exact opposit of names, it will override and log all ie 403, bad request. This way you can track how often or what is going wrong. "log_all_request=false" this will tell me to call morgan and log the requests. This way you can track what request is causing the actions, or maybe you want to track what is getting more traffic.

app.ts, blog.ts, role.ts src/app.ts loads the src/api/blog/blog.ts & src/api/role/role.ts for route handling along api routes. You will notice that code got smaller and more organized.

myTest is a controller that I use to test errors or other things that I want to run a quick test. You will see some commented out code that generates or creates errors, I even tell Typescript to ignore the errors //@ts-ignore. This way I could create 500+ errors which are server type errors and custom error handling for my log testing. This is a good place practice to place code that you want to test. If you leave it in your code you should put a production code detection, because you may not want someone to access this code if you for example (pre grant access and setup ready to do something). This file is src/controllers/mytest.ts and is callied in src/api/app.ts.

That is all there is.Modify the code and run the tests. Be sure to read the resources. There is more about securing your API and many other considerations.