Motiv Solutions

General Guidelines

Whenever possible code, solutions and applications should run in Docker. The idea is to develop directly in Docker. Running and Debugging should be happening in Docker.
Test Driven Development. The idea is to write a test FIRST before a class or function is created.
- UNIT tests should cover functions/methods that make sense to UNIT test
- Any bug found gets testing added for the case before a code fix
- All tests are expected to work with our Continuous Integration (CI/CD)
Once deployed/released, API’s must be immutable
- MotivLabs sees the importance of not breaking APIs our clients consume and code against
- We create new versions when a breaking change would be introduced. For example version 1 API vs 2 but Version 1 is immutable from the standpoint of tests, how to call it, and how it works.
- This is all enforced with both Integration Tests and UNIT Tests
- All APIs need Integration Tests. Integration Tests will test the whole stack ie.. a GO REST service backed by Cassandra etc…
MotivLabs engineers follow strict CI/CD processes. While MotivLabs might release often as we follow an iterative approach we ensure the release follows known and expected CI/CD processes
Documentation
- Code MUST be documented. A User Story should fail without proper tests and without Documentation
Code must conform to coding guidelines
- Ie.. For Go, that means formatted according to gofmt and following naming conventions
As a general rule in Go developers should avoid singletons. Variables should be created as close to possible to being used
- This will also help as our code is intended to be completely stateless. Singletons run the risk of storing things that require state and can lead to errors developers inadvertently make
Program to Interfaces. Program to Interfaces. Program to Interfaces.
All Repositories with REST Handlers should be tested in Postman and Documented in Postman
- In Postman we need a Documentation Collection for every Microservice
- This should be used to Test AND to Doc Rest End Points
- NOTE: all rest endpoints should return JSON: empty response should be a empty JSON () with proper JSON errors, codes and messages returned

Documentation Guidelines

Every repository should have it’s own documentation on how to use it

The README needs to cover the docker files and anything required to debug, install or config the code

Code for how to use the software makes its way to MotivLabs Documentation. Document according to the audience. Think about it this way

Code doc like running the code or coding the code goes in the README

Documentation for how to use the software goes in the Documentation for the software

REST Documentation goes in Postman

Each user story that adds, changes or updates functionality in ANY way should have documentation added

As a general rule of thumb every API which is defined as an End Point or anything someone could call should have the following.(This applies primarily to server side code)

Document all the APIs and how to use them. What is required? What is optional?

In Postman the errors should be examples. REST endpoints should have both the FULL and the Minimal doc provided with Markdown explaining how to use each endpoint and what the options are.

Documenting Code(This applies primarily to server side code)

In short summary a developer should not have to read the code to know how it works. All functions should document the following

The parameters takes. Can they be null what can be passed in?

What are the possible returns?

What errors are expected?

Unit Tests

Unit Tests should run fast and test business logic

A single unit test should run in a matter of seconds - not minutes

All unit tests for a component should run in 1 minute or less - speed is key here to enable quick iterations and fast code refactoring

Unit Tests run directly against the code being tested (not via remote call like REST or RPC)

In addition to public methods, unit tests can and should test functionality that is not necessarily public - e.g. private or non exported package functionality should be tested by unit tests

Unit tests should use the same programming language as the implementation code and should be positioned where it has access to private and non-exported package functionality.

While we will not test the router the method the router calls should be UNIT tested

Code dependencies should be mocked - e.g. no calls to actual database (this will encourage/require dependency injection capability to be in place)

This is an important point and Mock is a tough word. Our thinking is as stated in the guidelines above for Go we should not be using Singletons. This means no setter injection etc..

We do believe in programming to an Interface. So if a function writes to standard out the function should take the writer interface so you can test it

Unit Tests should be used as they make sense. For code that is related to CRUD for example it makes no sense to have a Unit Test but if the save logic has a validate() it makes sense to have that be its own function which can be Unit Tested

Trival code is a function that is just passing a call to another function. Like a function that only calls a DAO

Docker images should have a separate entry point that runs the unit tests

UNIT tests run in a container.

Tests should report properly results for both development and for automated CI/CD. All UNIT Tests MUST be named properly. We will use *_test.go

UNIT Tests should spit out a report. This means the Docker Container delivers an HTML Report

Integration Tests

Every single endpoint must have one or more integration tests

Tests the implementation along with any dependencies it may have - e.g. code and dependent services like databases, etc - nothing is mocked

Focuses on the external interfaces that will be called by others - all calls should be to the external/remote interfaces

All of the testing code and data must be external to the product code and container

E.g. microservice endpoint/method/API calls must be to the same endpoint/method/API that will be called in production - no separate test methods/endpoints in product code

Implementation should not contain logic to create, update, or delete test data.

Implementation being tested must be the same code that will run in production

E.g. implementation must not have code that is conditional on whether this is a test or not

Integration tests should be in their own docker images so that they can be run consistently in any environment

A docker compose file will be included in the project that will start the container(s) needed for the component being tested and a separate container that will run the integration tests.

For API Integration tests we will be using Newman to test

There are a number of good reasons Postman and Newman are good solutions for writing Integration Tests for APIs

It’s pretty simple to write test suites. There is less tooling out there then you would initially think. Postman has a GUI and the files it generates can be shared via source control. In addition to this you can use JS to test and assert on the responses

Postman Collections work have support for running in different environments. We can pass in URLs, domains etc… This is great as you can run in the GUI locally as a dev and allows easy automation through variables when scripting and running against different environments

Postman support storing data ie.. a JWT Token or even an ID to an object you created for a later call in the collection

Writing good integration tests

iTests need to test for

Expected errors. Ie.. if the code should 404 with a blank id then ensure it does and the proper documented message is returned

Make sure you test different files and scenarios

If the code works by supplying a host or without one test both cases

If you can pass 1 or more options test with only 1 and with multiple combinations.

Ensure all documented use cases have coverage

General guidelines

Ensure we use proper return code

An empty search should be 200 with an empty array

We need to return proper errors and messages in response

CRITICAL : The idea is to return as soon as possible to the client. We will return a transactionID but the lifetime of the transaction which will flow through Kafka and such should happen after we return. We need to be as ASYNC as possible and for sure need to avoid long running http threads. Once data is uploaded and in Kafka we return.

URIs

REST API’s should be designed for resources, which can be entities or services, etc., therefore, when possible, resource URIs should be based on nouns (the resource) and not verbs (the operations on the resource). For example, instead of /createUser use /users

In general, it helps to use plural nouns for URIs that reference collections. For example, /customers is the path to the customers collection, and /customers/5 is the path to the customer with ID equal to 5. The path of the endpoints that deal with nested resources should be done by appending the nested resource as the name of the path that comes after the parent resource. For example, if a user has posts and we want to retrieve a specific post by user, API can be defined as GET /users/123/posts/1 which will retrieve Post with id 1 by user with id 123.

Finally, it might not be possible to map every operation implemented by a web API to a specific resource. You can handle such non-resource scenarios through HTTP requests that invoke a function and return the results as an HTTP response message. Processing functions that don't modify data on the server but do perform some calculations can be abstracted by resources returning results for GET requests, for example GET /directions?from=Seattle&to=Portland can be used for a direction finder function or GET /validate?ccnum=1234567890123456 can be used to validate a credit card. On the other hand, actions that modify state or data on the server can be implemented by APIs that use POST, for example POST /alerts/245743/resend can be used to allow a client to resend an alert to a user.

HTTP Methods

The primary HTTP methods are POST, GET, PUT, and DELETE. These correspond to create, read, update, and delete (or CRUD) operations, respectively.

GET

Retrieves a representation of the resource at the specified URI. The body of the response message contains the details of the requested resource. Sending an HTTP GET request to an endpoint that represents a collection retrieves a list of items in the collection. An HTTP GET request to one the collection items URI returns the details of that item. For example GET /customers returns the collection of customers, and GET /customers/5 returns the details for the customer with ID equal to 5. GET requests over collection resources can potentially return a large number of items. To limit the amount of data returned by any single request, the API can support query strings that specify the maximum number of items to retrieve and a starting offset into the collection. For example you can use GET /orders?limit=25&offset=50

POST

Creates a new resource at the specified URI. The body of the request message provides the details of the new resource. POST request is applied over a collection, for example /customers , so the new resource is added to the collection. To create a resource using the full URI for the new resource, for example /customers/5 , you should use PUT method instead (in case the API supports that the client can specify the new resource ID). POST can also be used to trigger operations that don't actually create resources but that cannot be mapped to other core HTTP methods, for example it can be used to trigger a new job.

PUT

Either creates or replaces the resource at the specified URI. The body of the request message specifies the resource to be created or updated. If a resource with this URI already exists, it is replaced. Otherwise a new resource is created, if the server supports doing so (a server might support updates but not creation via PUT).

PUT requests are most frequently applied to resources that are individual items, such as a specific customer, rather than collections. Some API can support bulk HTTP PUT operations that can batch updates to multiple resources in a collection. The PUT request should specify the URI of the collection, and the request body should specify the details of the resources to be modified.

DELETE

The following table summarizes the above description of HTTP methods. Note that not all of these requests might be implemented: it depends on the specific scenario.

Summary

The following table summarizes the above description of HTTP methods. Note that not all of these requests might be implemented: it depends on the specific scenario.

Resource	GET	POST	PUT	DELETE
/customers	Retrieves all customers	Creates a new customer	Bulk update of customers (if implemented)	Removes all customers (if implemented)
/customers/1	Retrieves the details for customer 1	Not allowed	Updates customer 1 (or creates it if it's supported)	Removes customer 1

Other Methods

There additional HTTP methods that can be supported by APIs if required. The PATCH method performs a partial update of a resource. The request body specifies the set of changes to apply to the resource (PATCH is like PUT, but allows for fine-grained changes to resource state).

The HEAD method can be used to retrieve only the headers of the response, without the representation of the resource itself (the API returns an empty body). Clients can use this method to check whether a resource exists or to read its metadata.

The OPTIONS method can be used to discover which HTTP methods a resource responds to. The response includes an Allow header that includes the allowed HTTP methods for the resource.

HTTP Status Codes

HTTP status codes are divided into the following five categories:

1xx: Informational	These response codes are used only in negotiations between an HTTP client and server.
2xx: Success	Indicates that the client’s request was accepted successfully.
3xx: Redirection	Indicates that the client must take some additional action in order to complete their request.
4xx: Client Error	4xx: Client Error – The requested operation failed, due to a problem with the HTTP request (for example the request was malformed or invalid).
5xx: Server Error	The requested operation failed, due to a problem on the server side.

Common HTTP status codes for REST APIs include the following:

200 OK

It indicates that the REST API successfully carried out whatever action the client requested and that no more specific code in the 2xx series is appropriate.

201 Created

A REST API responds with the 201 status code whenever a resource is created inside a collection. There may also be times when a new resource is created as a result of some controller action, in which case 201 would also be an appropriate response. Also, if the action cannot be carried out immediately, the server should respond with a 202 (Accepted) response instead.

202 Accepted

A 202 response indicates that the request has been accepted for processing, but the processing has not been completed. This is an appropriate response when a request triggers an asynchronous action.

204 No Content

The 204 status code is usually sent out in response to a PUT, POST, or DELETE request when the REST API declines to send back any status message or representation in the response message’s body. An API may also send 204 in conjunction with a GET request to indicate that the requested resource exists, but has no state representation to include in the body.

301 Moved Permanently

The 301 status code indicates that the requested resource has been changed permanently. The new URL is given by the Location header field in the response.

302 Found

The 302 status code is used to indicate that the URL of the requested resource has been changed temporarily. The new URL is given by the Location field in the response.

303 See Other

A 303 response indicates that the API sends the client the URI of a response resource, instead of sending a potentially unwanted response body. The response can be the URI of the temporary status message, or the URI to some already existing, more permanent, resource.

304 Not Modified

Indicates the client that the response has not been modified, so the client can continue to use the same cached version of the response. The client can use headers If-Modified-Since or If-None-Match to indicate that it has a cached version of the resource, and is only interested in a newer version (modified) of the resource.

400 Bad Request

400 is the generic client-side error status, used when no other 4xx error code is appropriate. Errors can be like malformed request syntax or invalid request message parameters. For example, if an API is expecting a body in a JSON format for a POST request, but the body of the request is malformed.

401 Unauthorized

A 401 error response indicates that the client tried to operate on a protected resource without providing the proper authorization. It may have provided the wrong credentials or none at all.

403 Forbidden

A 403 error response indicates that the client’s request is formed correctly, but the user does not have the necessary permissions for the resource.

404 Not Found

The 404 error status code indicates that the REST API can’t map the client’s URI to a resource.

405 Method Not Allowed

The API responds with a 405 error to indicate that the client tried to use an HTTP method that the resource does not allow. For instance, a read-only resource could support only GET and maybe HEAD.

406 Not Acceptable

The 406 error response indicates that the API is not able to generate any of the client’s preferred media types, as indicated by the Accept request header. For example, a client request for data formatted as application/xml will receive a 406 response if the API is only willing to format data as application/json.

415 Unsupported Media Type

The 415 error response indicates that the API is not able to process the client’s supplied media type, as indicated by the Content-Type request header. For example, a client request including data formatted as application/xml will receive a 415 response if the API is only willing to process data formatted as application/json.

500 Internal server error

500 is the generic REST API error response. Most REST APIs respond with this response status code whenever they execute some request handler code that raises an exception.

501 Not Implemented

The server either does not recognize the request method, or it cannot fulfill the request. Usually, this implies future availability (e.g., a new feature for a REST API).

502 Bad Gateway

The server got an invalid response while working as a proxy or a gateway to get a response needed to handle the request.

503 Service Unavailable

This indicates that something unexpected happened on the server side (it can be anything like server overload, some parts of the system failed, etc.), so the server is not ready to handle the request.

Message Body Format

For now, our REST APIs use JSON for non-binary data. In the future, if other format like XML is required, a REST API can use Content-Type header to specify a different format (for example Content-Type: application/xml) for the message body, and also a client request could include an Accept header that contains a list of media types the client will accept from the server in the response message (for example Accept: application/xml).

For error messages, our REST APIs can include some of the following fields in the response body:

message	a brief message describing the error condition
error-id:	an identifier for the error
description	a longer description with information on how to fix the error condition, if applicable

Versioning REST APIs

Versioning enables a REST API to indicate the features and resources that it exposes, and a client application can submit requests that are directed to a specific version of a feature or resource. When a new version is released for a REST API, existing client applications should continue functioning unchanged while new client applications should be able take advantage of new features and resources for the new version. Our preferred approach for versioning is URI versioning: each time you modify the REST API, you add a version number to the URI for each resource. The previously existing URIs should continue to operate as before, returning resources that conform to their original schema. For example, GET /v2/customers/3 can be used to retrieve the customer with ID equal to 3, using the version v2 for the API.

Logging

We should log a lot. We should log a lot. We should have a lot of logging

A lot of logging doesn’t mean bad quality. Avoid the `I am here logs`

We should log a lot. We should log a lot. We should have a lot of logging

Logging should track what is happening and record the severity

Debug logs must be meaningful and must be used

There should never be functions without logging

Logging in a way that you will be the support engineer supporting the product

What to log examples

Date and time of the event in a standard timezone like UTC.

Session or Transaction ID once we get one

Data that will help us understand and make figure out what went wrong

error stacks, service name, header’s request, the URL path and its parameters, and similar kinds of data that will help you to reproduce the error and understand what happened.

Any Context data someone would need to debug a problem if it happened in production

The IP address of the server

Log levels such as panic, fatal, debug, or informational logs to give more context

Tracing

We use Opentracing and Jaeger to trace our microservices. We also have a Jaeger module that assists in injecting and extracting span context through kafka and HTTP. Standards are as follows.

Every microservice should have tracing.

Every module should have tracing.

Every possible function should have tracing.

When not possible, it is likely best to create a span and finish it right before and right after the function.

Spans should contain tags regarding:

There should never be functions without logging

Interface or package (Interface take precedence) If it’s a module.

It is a good idea to use span logs when something requires a timestamp.

I.e. the traceId for a kafka injection

The structure sent through a kafka message should include the trace Id.

The consumer of a kafka message should extract the trace Id.

There are many examples throughout all the services and modules of how to use tracing. We maintain a jaeger module that should be used as it will help inject and extract span context.

How to Program a Card/User Story

The following steps are the general order that should be followed when working on a card

If the card requires a REST Endpoint Postman Collection for it

Create the Function Stubs/Signatures. DO NOT CODE YET. RETURN NILL or leave blank but make sure they compile

Write the Documentation Collection with iTests

If it makes sense to have a UNIT test start to put it together or write the whole thing if possible

At this point the tests SHOULD be failing

Code the function/s and make ALL tests work and fill in missing test coverage and documentation

Code Review

We will ensure ALL cards get 2 Reviewers who must fill out the checklist and attach it to the Github Issue

Below is the template for the check list

Documentation

Are All Functions Documented

Is the Proper Doc in the Readme/Manual

Does Swagger have the right Model/Schema

Does Swagger have all required Examples

Are Status Codes right in Swagger

Are there Examples for all APIs ie.. CURL

Testing

Do all files have UNIT tests

Is Coverage 100% for UNIT Tests

Do the UNIT Tests cover all major cases

Do we have the proper coverage for Integration Tests

Logging

Are we providing proper debug logging

Are we providing proper info logging

Are we providing proper error logging

Is logging Contextual - (DAO fail vs API Fail)

Are there Security Concerns in Logging

Does the Log have Required Data to Debug (Helpdesk)

Code Quality

Are Packages Named Properly

Does the code do what it is supposed to

Are Patterns Followed According to Coding Standards

Summary

Testing Guidelines

Rest Endpoints

General Guidelines

Documentation Guidelines

Unit Tests

Integration Tests

General guidelines

URIs

HTTP Methods

GET

POST

PUT

DELETE

Summary

Other Methods

HTTP Status Codes

Common HTTP status codes for REST APIs include the following:

Versioning REST APIs

Logging

Tracing

How to Program a Card/User Story

Fixing a Bug

In addition to the How to Program a Card/User Story above here is what should happen when fixing or finding a bug

Code Review