cookie check in tntnet
Tntnet uses session cookies for keeping track of the client if there are variables defined in the <%session> scope. However, if you rely on clients accepting and sending cookies, you application might work erronous.
For example you could redirect the client to a login page, if there is no valid session set. If the client does not accept cookies, it would end in a redirection loop until the client detects the loop, stops sending requests and displays an inconvenient error page.
To fix that, you could enable some JavaScript to test if the client has cookies enabled, however this would not work for these clients, who doesn't have JavaScript enabled and also would most likely ignore exceptions the user might want to set just for you.
A better way is to check the cookie support with a redirect, here is an example for tntnet:
<%session scope="global">
bool cookies = false;
bool redirected = false;
std::string orig_query;
</%session>
<%cpp>
if(!cookies)
{
if(request.getQueryString() == "cookietest")
reply.redirect("/error/nocookie");
orig_query = request.getQuery();
cookies = true;
reply.redirect("/login?cookietest");
}
if (!redirected)
{
redirected = true;
reply.redirect(orig_query);
}
</%cpp>
First it checks if the cookie variable is set to false, if so it checks if the URL parameter is "cookietest" which the test would set later. If cookietest is the url parameter, it's safe to assume, that the browser has been redirected before and therefore will redirect the client to "/error/nocookie" which can deliver an appropriate error message
If the cookie variable is false and the url parameter is not "cookietest", it will store the original request in orig_query, set cookie to true and redirects to the testpage.
If the cookie variable is true, cookies are enabled and it's save to redirect to the original Page. To prevent a redirection loop when the original url will for what ever reasons cause the test again, we also set redirected to true
pentadactyl
I used Vimperator to have a vi like userinterface in firefox for many years and it was great. However, the latests developments in Vimperator i didn't like much.
Fortunately, some of the developers seem the agree with my understanding of a pleasant user experience and created Pentadactyl which feels just like Vimperator 2, but also with Firefox 7. Here's my .pentadactylrc
set autocomplete=0 set guioptions=Ccs set showtabline=never set noerrorbells
ecpp include
ecpp is the template language used in tntnet to join C++ with HTML. If you want to split your application into multiple ecpp files, you have two possibilities. You can either make it a subcomponent that you call using <& subcomponent>, or you can include the ecpp code using <%include>. Most people seem to prefere calling subcomponents, so if i prefere the other way, i should have good reasons.
If you create a subcomponent, you get an independent component that get called as a function. You can pass arguments to the component, but you don't have access to the scope of the calling component. The argument types are very limited you can only pass strings, or types that can be converted to strings. If your subcomponent takes arguments other than strings, ecpp will use tnt::stringTo<type> to convert the argument from a string to your type. The argument will always be a copy of a string, you can't pass by reference or a pointer to a string.
subcomponent example
So, if you want to use a subcomponent to printout parts from you own types, you need to pass the strings to the subcomponent. as example lets take a vector<string> and fill it with 100 strings. For printing we use a subcomponent called in a loop.
<html>
<head>
<title>include test</title>
</head>
<body>
<h1>main</h1>
<{
std::vector<std::string> vec;
for(int i=0; i<100; i++)
{
std::ostringstream s;
s << "string" << i;
vec.push_back(s.str());
}
}>
<h2>print a vector of 100 strings</h2>
% for(std::vector<std::string>::const_iterator it = vec.begin(); it < vec.end(); it++) {
<& comp arg1=*it >
% }
</body>
</html>
The subcomponent comp takes one string as argument, it is very simple.
<%args>
arg1;
</%args>
<p>argument: <$ arg1 $></p>
include example
The include example is very similar, but instead of calling a function to printout the string, the included ecpp becomes the body of the for loop. That saves a function call for each iteration of the loop, but more important, we don't need to copy the string to a different component.
<html>
<head>
<title>include test</title>
</head>
<body>
<h1>main</h1>
<{
std::vector<std::string> vec;
for(int i=0; i<100; i++)
{
std::ostringstream s;
s << "string" << i;
vec.push_back(s.str());
}
}>
<h2>print a vector of 100 strings</h2>
% for(std::vector<std::string>::const_iterator it = vec.begin(); it < vec.end(); it++) {
<%include>
incl.ecpp
</%include>
% }
</body>
</html>
The included file is even more simple. Notice, that it prints out whatever it points to, there is no defenition what type it is, so it really depends on what file includes it. This way it is possible to use the same include file for may different ecpp components with differnet types, as long as the objects in the components have the same name used in the include file and have all required operators and members.
<p>outstr: <$ *it $></p>
Comparing
When talking about performance we should first consider if it does matter. To find out how large the overhead of the subcomponent variant is, i used siege to compare both components, the first, mapped to /comp, uses the subcomponent printout, the second uses include.
$ ./siege -c 100 -b -r 1000 'http://127.0.0.1:8000/comp' ** SIEGE 2.70 ** Preparing 100 concurrent users for battle. The server is now under siege.. done. Transactions: 100000 hits Availability: 100.00 % Elapsed time: 260.25 secs Data transferred: 35.95 MB Response time: 0.25 secs Transaction rate: 384.25 trans/sec Throughput: 0.14 MB/sec Concurrency: 97.20 Successful transactions: 100000 Failed transactions: 0 Longest transaction: 24.58 Shortest transaction: 0.00
$ ./siege -c 100 -b -r 1000 'http://127.0.0.1:8000/incl' ** SIEGE 2.70 ** Preparing 100 concurrent users for battle. The server is now under siege.. done. Transactions: 100000 hits Availability: 100.00 % Elapsed time: 112.77 secs Data transferred: 33.95 MB Response time: 0.11 secs Transaction rate: 886.76 trans/sec Throughput: 0.30 MB/sec Concurrency: 94.05 Successful transactions: 100000 Failed transactions: 0 Longest transaction: 13.64 Shortest transaction: 0.00
So the difference is quite large, i got more than twice as much requests/sec done when using the include variant. Of course the example application is very simple, but i do use small ecpp snippets all the time to print a few strings from my objects. If you for example what to share the HTML headers for all sites but want to change <title>, it would become much faster, if you use include instead of passing the title string to a subcomponent.
Also i take advantage of the fact that include is type agnostic, if i just like to call object.getText() with some surrounding HTML, it really doesn't matter, what type the object is, as long as it have a member funktion getText() that returns something that can be converted to a string.
Subcomponents also have it's application, if you have larger subroutines that acts independent from the calling component and is called by multiple components, a subcomponent is typically to prefere. The overhead of calling a subcomponent especially without parameters is not that large and you can benefit from a smaller binary size, since a subcomponent get only included once in the whole application.
readline vi-mode
If you like to use your shell in vi mode, it's very likely you also love to have a vi-mode in psql and mysql or just everywhere. This is achievable if the tool of your choice can be compiled with readline support. Just check ldd if it is the case:
$ ldd `which psql` | grep readline
libreadline.so.6 => /lib64/libreadline.so.6 (0x0000003ad1000000)
If so, you can enjoy using vi-mode in psql by enabling the readline vi-mode:
echo 'set editing-mode vi' >> ~/.inputrc
This enables vi-mode for all commandline tools linked against readline. Interestingly it also enables a more advanced tab completion in mysql. DESCR<tab> will now expand to DESCRIBE for example.
Singleton template instantiation
When you create a sigleton, you want to have exact one instance in your programm. But what does happen if you have a singleton template that get instantiated multiple times. I did a simplified test to show what happens.DeleGate
DeleGate is just amazing, it is not only a very simple HTTP and SOCKS proxy, it also can redirect TCP Ports to a host in an other network or provide the content of a /etc/hosts file as DNS Server and much more. Here are a fews usage examples.
Simple HTTP reverse Proxy, caching behaviour can be controlled optionally
delegated -P80 SERVER=http://www.vtoc.de/
Forwarding a TCP (Postgres in this case) Port. This can be usefull, if you migrate the database to an remote host and you canot take the ip with you. An IP after -P tells delegate to bind that address only
delegated -P10.20.20.42:5432 SERVER=tcprelay://172.16.20.23:5432/
Tunnel a TCP connection thru a socks proxy
delegated -P10.20.20.42:5432 SERVER=tcprelay://172.16.20.23:5432/ SOCKS=proxyhost:port
Making an HTTPS only service available to HTTP clients
delegated -P80 SERVER=http MOUNT="/* https://httpshost/*" STLS=fsv:https
SSL proxy for ftp
delegated -P990 SERVER=ftps MOUNT="/* ftp://localhost/*" STLS=fsv
Standalone HTTP Server
delegated -P80 SERVER=http MOUNT="/* file:/var/www/html/*"
DNS Server from hosts file
delegated -P53 SERVER=dns RESOLV=file:/etc/hosts
DeleGate can do much more, like cronjobs or run as inetd server, best you look into the documentation.
Building gcc4 with gcc3
Compiling gcc4 on an system with only old gcc versions available, can be tricky. Here is an HowTo for compiling gcc4.4 on Solaris 10, which only ships 3.4.3. It will be similar for other systems with an older gcc versions.Monitoring Postgres autovacuum
Postgres does all updates on a database Copy-on-Write. That means, on every update on a row, additional space is needed on the filesystem. If noone is using an old entry anymore, the space allocated by these data is not freed automatically, nor will it be available for postgres to store new data. To get the space back, you have to vacuum, luckily there is an autovacuum in postgres. Autovacuum is an extra deamon that will run vacuum from time to time. It will not run a vacuum full, thus it will never free disk space, but postgres is able to reuse the already allocated space and thus the datafiles will not grow infinitely.
The cumbersome part is to check if autovacuum is working, since there is no way to ensure that it will do. First you can check, if autovacuum is configured to run. You might think setting "autovacuum = on" would be enough, it's not, since it will only work with some version depending statistics enabled.
Potstgres >=8.2
SHOW autovacuum; SHOW track_counts;
Prior versions:
SHOW autovacuum; SHOW stats_start_collector; SHOW stats_row_level;
Next is to check, if the autovacuum daemon is running. There is one postgres process that will be named "postgres: autovacuum launcher process". If you are running solaris, ps will not show you that name, instead it will show the original process cmdline, since it is ignoring updates on argv[0] made by processes at runtime. You can help your self with a little loop.
for PGPR in `/usr/bin/pgrep postgres`; do pargs -a $PGPR; done | grep 'argv\[0' | sed 's/ *$//'
An more easy and generic way is to look into the logfile, if you've not already deleted the log containing the startup messages, you'll find the following line there:
LOG: autovacuum launcher started
You can also monitor the activity of autovacuum on the database. However, it will only show you activities if autovacuum had done something, thus on an fresh installed database it is very unlikely to see any activity.
postgres=# SELECT relname,last_autovacuum from pg_stat_user_tables;
relname | last_autovacuum
-------------------------+-----------------
my_table | 2011-05-02 05:05:11.78106+02
my_other_table | 2011-05-03 09:21:18.850887+02
unlikely_for_update |
If you select pg_stat_all_tables instead of pg_stat_user_tables you will also see vacuums for internal tables.
drace syscall times
Sometimes processes are slow without consuming much ressources. In such cases you can try to use truss (or strace on some systems) to find out what the process is waiting for, however, in multithreaded processes this might not help you that much. Also tracing the process from userland will slow down the process further what might not be an option in production einvironments.
Using dtrace will not only be less intrusive to your application flow, it can also summarize the times for you. The following script will not only summarize the times spending in system calls (/opt/DTT/procsystime would do), it will also print you a line for each system call when it retruns
#!/usr/sbin/dtrace -s
#pragma D option quiet
syscall:::entry
/ zonename == "myzone" && execname == "tntnet" /
{
self->t=timestamp;
}
syscall:::return
/ zonename == "myzone" && self->t!=0 && execname == "tntnet" /
{
printf("%d/%d spent %d nsecs in %s\n",
pid, tid, timestamp - self->t, probefunc);
@function[probefunc] = sum (timestamp - self->t);
}
For these who are familiar with awk, the syntax will not surprise you, in / / there are filters on which the code blocks apply, self is a special variable you can use to ensure coherency between threads.
... 14611/52 spent 3415416747 nsecs in pollsys 14611/52 spent 8583 nsecs in gtime 14611/52 spent 5333 nsecs in gtime 14611/52 spent 21666 nsecs in lwp_park 14611/52 spent 2798833 nsecs in nanosleep 14611/52 spent 20010167 nsecs in pollsys 14611/52 spent 18000 nsecs in read 14611/52 spent 9864917 nsecs in nanosleep 14611/52 spent 5417 nsecs in gtime 14611/53 spent 10008364573 nsecs in lwp_park 14611/53 spent 11500 nsecs in gtime 14611/53 spent 5500 nsecs in gtime ^C lseek 29583 fcntl 34666 close 45083 accept 57417 getpid 133751 open 134834 stat 248915 gtime 282585 read 1725499 write 2817832 nanosleep 80435583 pollsys 5699637580 lwp_park 35622920131
You see, that the process is spending the most time on beeing idle and waiting for requests, but tntnet is actually not a really good example since it doesn't appreas to be slow.
Solaris ptools
Solaris has some features, many other systems are missing. One of my favorites is the large number of process tools. While i could write the whole day about ptools, i like to concentrate on my favorites here. Some of them may also be are available on other plattforms.
pldd
pldd acts like ldd but on processes, thus you see which librarys are really loaded by the program. Yes there might are differences to the binaries on the filesystems.
$ pldd 1259 1259: /home/jw/bin/ncmpc /usr/lib/libglib-2.0.so.0.2501.0 /lib/libsocket.so.1 /lib/libnsl.so.1 /usr/gnu/lib/libncurses.so.5.7 /usr/lib/libc/libc_hwcap1.so.1
pargs
pargs shows arguments -a and environment variables -e of an running process.
$ pargs -ea 284 284: /usr/lib/inet/ntpd -p /var/run/ntp.pid -g argv[0]: /usr/lib/inet/ntpd argv[1]: -p argv[2]: /var/run/ntp.pid argv[3]: -g envp[0]: _=*213*/usr/lib/inet/ntpd envp[1]: LANG=en_US.UTF-8 envp[2]: PATH=/usr/sbin:/usr/bin envp[3]: PWD=/root envp[4]: SHLVL=1 envp[5]: SMF_FMRI=svc:/network/ntp:default envp[6]: SMF_METHOD=start envp[7]: SMF_RESTARTER=svc:/system/svc/restarter:default envp[8]: SMF_ZONENAME=global envp[9]: TZ=Europe/Berlin envp[10]: A__z="*SHLVL
pstack
pstack prints a strack trace from an running process.
$ pstack 11717 11717: ksh ceb04875 waitid (7, 0, 8046f70, f) ceaa00a5 waitpid (ffffffff, 804703c, c, ce99076f) + 65 ce9907c8 job_reap (0, ce9f7000, 80470a8, ce99287c) + c0 ce992af7 job_wait (3c82, 806a8a8, 0, 8047130) + 4ab ce9c8ae3 sh_exec (806755c, 4, 0, ce9a0f3d) + 43af ce9a10e6 exfile (ce9f7bb0, ce8d0fa8) + 732 ce9a0920 sh_main (1, 8047964, 0, cea5ea42, 4, 0) + 830 08050bd6 main (1, 8047964, 804796c, 8050a9f) + 4a 08050afd _start (1, 8047a7c, 0, 8061210, 8061230, 8061248) + 7d
plimit
plimit works like ulimit on processes. Remarkable is, that it doesn't only print the limits of an running process, you can also modify the limits without restarting the process.
$ /usr/bin/plimit 11716 11716: xterm resource current maximum time(seconds) unlimited unlimited file(blocks) unlimited unlimited data(kbytes) unlimited unlimited stack(kbytes) 8480 130336 coredump(blocks) unlimited unlimited nofiles(descriptors) 256 65536 vmemory(kbytes) unlimited unlimited $ /usr/bin/plimit -n 1024 11716 $ /usr/bin/plimit 11716 11716: xterm resource current maximum time(seconds) unlimited unlimited file(blocks) unlimited unlimited data(kbytes) unlimited unlimited stack(kbytes) 8480 130336 coredump(blocks) unlimited unlimited nofiles(descriptors) 1024 65536 vmemory(kbytes) unlimited unlimited