Course outline · 0% complete

0/27 lessons0%

Course overview →

Connecting from EC2

lesson 6-2 · ~10 min · 17/27

Endpoints, not IPs

RDS gives your database an endpoint, a DNS hostname like:

myapp-db.c9xkzq2b1f.us-east-1.rds.amazonaws.com  port 5432

Your app connects to it exactly as it would to any PostgreSQL host, no AWS-specific driver:

psql -h myapp-db.c9xkzq2b1f.us-east-1.rds.amazonaws.com -U appuser -d myapp

Always configure the hostname, never the IP behind it. During a Multi-AZ failover (lesson 6-1) the hostname is repointed at the standby, so apps that use the endpoint recover automatically, and apps that cached the IP stay down.

before failover after failover app servers db.abc.rds.amazonaws.com primary us-east-1a standby, idle us-east-1b resolves to 10.0.2.15 app servers db.abc.rds.amazonaws.com gone us-east-1a promoted us-east-1b same name, now 10.0.3.27
The endpoint is a name, not a number. Failover repoints that same name at the promoted standby, so an app that re-resolves the hostname recovers and an app holding a cached IP keeps dialing a machine that is gone.

Why the hostname rule matters

The endpoint is a DNS record from lesson 5-3 that RDS controls. During failover, RDS repoints it at the standby's IP, and this simulation compares two apps.

resolve() {
  if [ "$1" = "before" ]; then echo "10.0.2.15"; else echo "10.0.3.27"; fi
}
cached_ip=$(resolve before)
echo "before failover, the endpoint resolves to $(resolve before) (the primary)"
echo "-- primary AZ fails, RDS repoints the endpoint at the standby --"
echo "app that re-resolves the hostname: connects to $(resolve after), recovers"
echo "app that kept the cached IP: dials $cached_ip, connection refused"

Output

before failover, the endpoint resolves to 10.0.2.15 (the primary)
-- primary AZ fails, RDS repoints the endpoint at the standby --
app that re-resolves the hostname: connects to 10.0.3.27, recovers
app that kept the cached IP: dials 10.0.2.15, connection refused

Both apps have correct credentials and correct permissions. The only difference is whether they stored a name or a number, and that difference decides which one survives the failover.

The failure mode is also quiet in a specific and unhelpful way. The app that cached the IP does not report a DNS problem or a permission problem, it reports connection refused, which sends people looking at security groups instead of at their own configuration.

Security group chaining

The database sits in a private subnet, and its security group needs an inbound rule for port 5432. What source? You could enter the app servers' IP range, but servers come and go, and their IPs change.

The idiomatic answer, promised in lesson 3-3: use another security group as the source.

security groupinbound rule
sg-app (on app servers)443 from the ALB's group
sg-db (on the database)5432 from sg-app

Read the second row as: port 5432, from any machine that wears sg-app. Launch a third app server tomorrow, give it sg-app, and it can reach the database with zero rule edits. Membership in the group IS the credential.

Why one legacy service stays broken after a failover

The likely cause is that the legacy service resolved the hostname once, cached the old primary's IP, and keeps dialing a dead machine.

Failover works by repointing the DNS name at the standby. Anything that keeps using the old IP is dialing a machine that is gone, and no amount of retrying will help, because retrying the wrong address forever is still the wrong address.

There are two ways a service ends up in that state. Someone put the resolved IP in a config file, or the client library resolved the name once at startup and never looked again.

The fix is to configure the hostname and keep DNS caching honest. The record's TTL from lesson 5-3 is how many seconds a resolved address may be reused, and a client that ignores it keeps serving itself the dead IP long after the record changed. Connection pools are a common culprit, since a pool that never expires idle connections also never re-resolves.

The right source for the database rule

The source should be the app tier's security group, sg-app.

0.0.0.0/0 means the whole internet, which violates least privilege from lesson 2-1 even inside a private subnet. The subnet's lack of an internet route contains the damage, as lesson 5-2 covered, but relying on that is trusting one layer to cover for a mistake in another.

Referencing sg-app grants exactly the machines that should connect, and it tracks them automatically as servers are replaced. Launch a fourth app server tomorrow, give it sg-app, and it can reach the database with no rule edit at all.

The mental shift is worth naming: the source is not an address range, it is membership in a group. Wearing sg-app is the credential, and that is why the rule keeps being correct as the fleet changes.