AWS supports IAM Roles to authenticate with RDS instead of conventional username password. We can create IAM policies for RDS and attach it to an EC2 instance and the EC2 will be able to connect with RDS using an IAM token instead of the password. In this article we will talk about how to periodically refresh the token when using pgx as the postgres driver in go. Or how to use/refresh IAM tokens when using django and python.
In this blog I will not go into the details of creating and attaching an IAM role/policy, there are a lot of resources available for this.
With IAM database authentication, you use an authentication token when you connect to your DB instance. An authentication token is a string of characters that you use instead of a password. After you generate an authentication token, it's valid for 15 minutes before it expires. If you try to connect using an expired token, the connection request is denied.
Network traffic to and from the database is encrypted using Transport Layer Security (TLS).
Using this authentication we can avoid keeping the password in the code/config. The AWS SDK will programmatically create and sign an authentication token.
The AWS SDK for go in case of Golang or boto3 in case of Python is used to generate a new token every time a db connection needs to be made. Important thing to note is that the token is generated on the client side. No network call is made to fetch the token from AWS. So it is completely okay to generate a new token every time a new connection is made.
How this is achieved is using AWS's Signature v4 signing process. AWS SDK signs the token with the access key ID and the secret access key.
We use this token to connect to Postgres instead of the password.
AWS has quite good documentation for connecting with Postgres/RDS using IAM tokens instead of passwords in go as well as in python. But when I was trying to implement this, the problem I faced was that we used client side database pools. It would be fair to assume that most real-world production systems leverage pooling.
Database connection pooling is a method used to keep database connections open so they can be reused by others.
Typically, opening a database connection is an expensive operation. You have to open up network sessions, authenticate and so on. Pooling keeps the connections active so that, when a connection is later requested, one of the active ones is used in preference to having to create another one.
Now this adds an additional layer of complexity. The pools are created lazily. That means if a pool has 50 connections, all the connections are not stood up when the app starts, the connections are added to the pool as and when queries/transactions arrive. Now if we were to reuse the IAM token, we would have to keep track of them. So we would need a timer to refresh the token every 15 mins. But we cannot really close the connections within a pool as these are self-managing pools. So we would have to close the pools and re-initialize everything again. You can see where I am going with this. This approach seems very hacky and is not elegant.
Go implementationFortunately pgx has recently added a BeforeConnect hook to its API. We can use this hook to modify the db credentials before any connection is made. This way every connection in the pool will have its own IAM token and as established earlier, tokens are created on the client side so there is no network call to "fetch" the token.
But how do we refresh the token after 15 mins? We don't! We can set the connection lifetime to 14 mins, so the connection is closed after 14 mins and the new connection that comes up, is created using a new token. This way we can inject our token for every connection. This approach also works because we are not closing the complete pool, we only close one connection at a time (because the connection lifetime is per connection). And because each connection is established lazily and has its own token, the connections are created and closed at different times, so the pool should always have some connections for use.
// peer contains the connection pool
// IAMRoleAuth is the flag to toggle IAM role based authentication
type peer struct {
name string
dbPool *pgxpool.Pool
weight int
logger log.Logger
mu sync.Mutex
IAMRoleAuth bool
}
type DBConfig struct {
Host string
Port int
User string
Password string
SSLMode string
Name string
MinConn int
MaxConn int
LifeTime string
IdleTime string
LogLevel string
Region string
IAMRoleAuth bool
}We create a session struct, and pass this on to create a new IAM token.
import "github.com/aws/aws-sdk-go/aws/session"
opts := session.Options{Config: aws.Config{
CredentialsChainVerboseErrors: aws.Bool(true),
Region: aws.String("us-east-1"),
MaxRetries: aws.Int(3),
}}
sess := session.Must(session.NewSessionWithOptions(opts))// getDBPool returns a new pgxpool.Pool instance.
// If peer IAMRoleAuth is true then BeforeConnect method is implemented
// BeforeConnect() is used to inject the authToken before a connection is made.
// Connection `LifeTime` is set to 14 mins, hence the connection will expire
// automatically and no intervention is needed to close the connection.
func (p *peer) getDBPool(ctx context.Context, cfg DBConfig, sess *session.Session) (*pgxpool.Pool, error) {
poolCfg, err := pgxpool.ParseConfig(getDBURL(cfg))
if err != nil {
p.logger.Err(err).Msgf("unable to parse config for peer: %v cfg: %v", cfg.Name, cfg)
return nil, err
}
if p.IAMRoleAuth {
poolCfg.BeforeConnect = func(ctx context.Context, config *pgx.ConnConfig) error {
p.logger.Info().Msg("RDS Credential beforeConnect(), creating new credential")
newPassword, err := p.getCredential(poolCfg, cfg, sess)
if err != nil {
return err
}
p.mu.Lock()
config.Password = newPassword
p.mu.Unlock()
return nil
}
}
pool, err := pgxpool.ConnectConfig(ctx, poolCfg)
if err != nil {
p.logger.Err(err).Msg("unable to connect to db")
return nil, err
}
return pool, nil
}
// getCredential returns the new password to connect to RDS
func (p *peer) getCredential(poolCfg *pgxpool.Config, cfg DBConfig, sess *session.Session) (string, error) {
dbEndpoint := fmt.Sprintf("%s:%d", poolCfg.ConnConfig.Host, poolCfg.ConnConfig.Port)
awsRegion := cfg.Region
dbUser := poolCfg.ConnConfig.User
authToken, err := rdsutils.BuildAuthToken(dbEndpoint, awsRegion, dbUser, sess.Config.Credentials)
if err != nil {
p.logger.Panic().Err(err).Msg("Error in building auth token to connect with RDS")
return "", err
}
return authToken, nil
}If you are using sqlx as the DB driver, then things get trickier. As sqlx does not provide the BeforeConnect() hook to inject the token. I have not found any elegant way to do this with sqlx yet.
Django implementationDjango is a very popular web framework in python.
Unlike the go implementation, django does not use database pools, instead django uses persistent connections. This is different from pooling because django maintains one persistent connection per thread. In pooling, the pool takes care of opening and closing connections and every thread that needs a connection requests one from the pool.
django uses postgres database backend which internally uses Psycopg as the database adapter. We can create our custom database backend by extending the postgres backend.
def get_aws_connection_params(params):
"""
"rds_iam_auth" : If set to `True` IAM authentication is used,
else password based authentication is used
"region_name" : Contains the name of the aws region where DB is present.
Parameters
----------
params : dict
The DATABASES dict that is passed from the settings.py
"""
enabled = params.pop("rds_iam_auth", False)
if enabled:
region_name = params.pop("region_name", None)
rds_client = boto3.client(service_name="rds", region_name=region_name)
hostname = params.get("host")
hostname = hostname if hostname else "localhost"
params["password"] = rds_client.generate_db_auth_token(
DBHostname=hostname,
Port=params.get("port", 5432),
DBUsername=params.get("user", getpass.getuser()),
)
return params
class DatabaseWrapper(base.DatabaseWrapper):
def get_connection_params(self):
params = super().get_connection_params()
params.setdefault("port", 5432)
return get_aws_connection_params(params)In the settings.py we can use this backend as:
DATABASES = {
"default": {
"ENGINE": "path.to.backend.aws.postgres",
"NAME": "postgres",
"USER": "user",
"PASSWORD": "welcome",
"HOST": "localhost",
"PORT": "5432",
"OPTIONS": {
"rds_iam_auth": True,
"region_name": "us-east-1",
"sslmode": "verify-full",
"sslrootcert": "/path/to/rds-ca-2019-root.pem",
},
},
}Another thing to note in both the above approaches, is that the hostname needs to be used to connect with the RDS PostgreSQL, if a DNS CNAME is used, the connection will fail with PEM authentication error. This happens because the IAM token is signed for the CNAME, but RDS expects the hostname. One work-around is to resolve the CNAME into the hostname in the custom backend, before making the connection. I would advise against this, because CNAMEs are cached at multiple places and have a certain TTL, the connection would fail if the CNAME has not propagated yet.
A term insurance is a very simple insurance where a person is insured for a "term" (say 30 years). You are required to pay the premium for this duration. During the term if you pass away, then your nominee receives the covered sum. However if at the end of the term if you are still alive, you don't get back anything. The premiums of these insurance policies are lower if you purchase them at a relatively young age. The premiums are fixed throughout the term of the policy, so it makes sense to buy these when young.
There are some people with another school of thought who say that you should get a term insurance only after you have dependents. This is also okay, but I personally would recommend getting one as soon as possible.
Now why term insurance is the right insurance for most people and how keeping your investments and insurances separate is the right thing to do is a discussion for a separate article. If you proceed to read further I assume you know why a term insurance is better than a ULIP.
In this article I will talk about how I purchased my term insurance and what are the things one needs to look out for.
It is usually recommended that the policy cover should be 10-20x your annual salary. I personally agree with this, the rationale behind this is that in the event of your death your family must be able to maintain their current lifestyle for the next 10-20 years. We also need to understand what "term" we want our insurance for. The idea is simple, we want the insurance till the time we are earning and we have dependents. So for most people I believe that would be the age of 58-60. By then most people would retire and their children would be self sufficient. There is no point of having an insurance if you aren't working and don't have any dependents.
Insurance plans also come with a load of riders like, Critical Illness Rider and Accidental Death Rider and a few others. In my opinion one should opt for a vanilla insurance as most of these riders are useless. Because we should have a separate health insurance for these things anyway.
For this, I just used policybazaar.com. It gives an easy way to look at the estimated premiums and claim settlement ratios etc on a single page. You can filter and sort based on the parameters important to you. While buying my policy I observed that most of the companies have claim settlement ratios etc greater than 97-98%. But one thing to keep in mind is that these claim settlement ratios are for all their insurance policies, which include ULIPs etc which are usually for low amounts. An insurance company can very easily approve 99 ULIPs with a cover of 5 lakh and reject 1 term insurance with a cover of 20 crore. As of now there really is no published data to look into this.
But the bottom line is that you can go with any major insurance company as you don't want your nominee to go through trouble if they ever need to file for a claim. In my case I chose ICICI Prudential. I chose my term of 38 years and looked at the premium and other policy details. Policy bazaar said that there would be a "Tele medical" before this policy was issued.
I had seen many reviews of policy bazaar saying that it was taking longer time to issue a policy through policy bazaar than buying directly from the company. So I decided to purchase directly from ICICI Pru.
I created an account on the insurer's website, filled the required details. Needless to say, don't lie on your forms as your claim might get rejected. It is fine to pay some extra premium but don't hide any medical history. For the insurance I would recommend just paying the premiums annually, against opting for any kind of limited pay. I paid the premium online. Yes, I paid the premium before the medical was done. It is possible that if there are some issues in the medical report, you might have to pay the extra premium. At this stage I was told that tele medical is not available so there will be a physical medical and sample will be collected from home.
I received a message from the medical partners of ICICI where I was asked to choose a time slot for the medical test. Here when I input my address I was prompted that home collection is not available in this area. This was very weird because I live in a very well known locality in North Bangalore. Since I had already paid the premium, I decided to go ahead with the medical. The lab was just 5 minutes from my house.
I went in at the selected time, everything was very smooth. I gave all my samples.
Within 48 hours, ICICI sent me the reports which is a great thing, because I've heard that many insurers don't send the medical reports back to the customers.
Within 2 days after receiving my medical reports, I was issued the policy. I received the policy documents over email on the same day. And within 14 days I received the physical policy documents. I was able to login to their website using my email and could verify that all the details were correct.
All in all the process was not very difficult, having said that, it wasn't the easiest. Tele medical or home sample collection would've been great. If you don't have a term insurance yet, I'd suggest you get one sooner rather than later.
Everything mentioned in this article is my opinion and is from personal experience, your mileage may vary.
If you'd like to know more, do check out:
]]>When I started working as an intern, I had the luxury of meeting people physically, going on team lunches etc. I could just walk over to someone if I had a question. These things really helped me build very good working relationship with my teammates, manager etc. However, with most offices still closed and most job opportunities becoming remote, it becomes all the more important to reach out and seek for help/feedback and make personal connections with your colleagues.
This article is based on my experience over the past 3 years, most things on here are just my opinion and its totally okay if you don't agree.
When transitioning away from academics one needs to understand the paradigm shift. Some things which were important in university are not very important anymore, while some other things like readability of code are the most important thing. Because of this, a lot of students struggle transitioning from academia to industry. In my experience the best students don't always make the best engineers. Students are typically used to working alone and are not used to taking feedback. In most universities the projects are delivered at once and students are not used to working on massive projects and making incremental progress.
Don't be afraid of taking initiative. Volunteer for new projects, volunteer if anyone needs some help. This will open up new opportunities for learning. It is okay if you don't know how to do something, you can always learn.
I have seen most interns/junior engineers don't participate in technical discussions. I believe the reason for this is that they believe that they are not good enough and don't want to sound stupid. But you need to not worry about sounding stupid and if you have a question or a suggestion, you should almost always speak up.
NO ONE will judge you for not knowing something. However please don't ask something just for the sake of asking, that is just annoying.
Taking ownership means that you will complete your assigned task from start to finish. You will ensure that you communicate with your team and the respective stakeholders in case there is any blocker or to keep everyone up to date on the status of the work, so that you reduce the risk of not getting your task done in time. This is something that junior engineers actively need to work on.
I have seen so many juniors who when given some task, just vanish and go into a cocoon for several weeks and come back after a few weeks having completed the work, or to tell that they are stuck with a problem. Taking ownership means that you need to show incremental progress and seek for feedback periodically. This will also help you develop trust with other team members as people will know what you are actively working on.
I cannot remember where I had read "If your manager cannot trust you, it doesn't matter how smart you are". This makes complete sense, you need to build trust with your co-workers. You need to ensure that people don't have to follow up with you again and again.
If you need something say it. If you need technical guidance, speak up. If there is something in your personal life that is taking up your time, it might make sense to share relevant information with your manager so that expectations can be set accordingly. If you have concerns regarding your promotion or your compensation, you should be able to talk to your manager about these things. Clear communication is very important.
A lot of junior engineers that I talk to, say things like "I'm not good at front-end" or "I don't know about XYZ". Well, that might be true for the moment, but you can always learn whatever you don't know. Most people don't know most things. It is more important to be open to learning. Learning new things itself is a skill. This brings me to my next point.
The best engineers I know are open to learning. They are often faster at learning than "others". This is not because they are smarter, but this is because they have developed this skill of learning new things over the years. Everyone has a different way of learning, figure out what works for you.
Usually you'll have to go outside your comfort zone for this. But the more new things you learn, the more you exercise your "learning muscle". Be open to suggestions and new ideas. These can be in the form of code review or during stand up meetings. Just general discussions between other team members is a great learning opportunity. Listen carefully to what others are saying, if senior engineers are doing something a particular way there is almost always a reason they do it that way.
If you don't know something, ask. No one will look down on you for not knowing something. And if they do, it is a reflection on them and not on you. It is better to ask a senior engineer who can explain something to you in 5 mins rather than spending 3 days trying to figure it out. However, please make a genuine effort to understand something before you go out and ask. You need to respect people's time. I see far too many junior developers who seek help at the first sight of inconvenience. This ties together with my previous point, if you don't struggle, you will not learn anything new.
Learn to switch off. This is something even I struggle with. You don't need to be online 24/7. If you receive an email at 11 PM you don't need to reply instantly. You don't need to work weekends. A lot of new engineers want to "prove themselves", which is fine. But you need to create a sustainable work-life balance for yourself, it is not fun to feel burnt out in a couple of years.
Having said that the other extreme is bad too. I've seen people not joining important meetings which are beyond 5PM because they are "off work", try to find the right balance that works for you.
Human beings are social creatures. Good relationship with our co-workers will make our jobs enjoyable. Be empathetic towards your teammates, try to help wherever you can. Having a strong professional circle will also help you develop your career, opening up opportunities that otherwise might pass by you.
Most of these things need to be done within reason, for example too much "asking" will come across as inability to learn or by taking up too many tasks you can stretch yourself too thin.
These were some of the things that I did as a junior engineer which helped me grow as an engineer. I still try to actively do these things. If you feel that I missed out on something or if there is something that you would do differently, please let me know.
]]>[Add your training leading up to the half marathon, travel to Mumbai, race day morning preparation]
[Describe the IIT Bombay campus course - the scenic lake, hilly terrain, multiple loops, atmosphere]
[The first third - settling into pace, how did you feel starting out?]
[The middle section - maintaining rhythm, any challenges?]
[The final push - how did you handle the last kilometers? Mental game?]
Replace the activity ID below with your actual Strava activity ID:
[Your reflections - achieving goals, what you learned, would you run it again?]
[Any advice for people planning to run this event]
Have questions about this race? Feel free to reach out on Twitter.
]]>[Describe what the Pinkathon is about - its cause, the atmosphere, the community]
[Add your pre-race preparation and expectations]
[Describe your race experience - the start, middle, and finish]
Replace the activity ID below with your actual Strava activity ID:
[Your reflections on the event and the experience]
Have questions about this race? Feel free to reach out on Twitter.
]]>[Add your pre-race preparation, training leading up to the event, and how you were feeling before the race]
[Describe the course - the route through Bangalore, elevation, atmosphere of running at midnight, crowd support, aid stations]
[How did the race start? What was the atmosphere like at midnight?]
[The middle section - how were you feeling? Did you stick to your pace?]
[The final push - did you have a strong finish?]
Replace the activity ID below with your actual Strava activity ID:
[Your reflections - what went well, what would you do differently, overall experience of the midnight run]
Have questions about this race? Feel free to reach out on Twitter.
]]>