firstworks Programming with SQL Relay using the Python DB API

Programming with SQL Relay using the Python DB API

- Establishing a Session
- Executing Queries
- Commits and Rollbacks
- Temporary Tables
- Catching Errors
- Bind Variables
- Re-Binding and Re-Executing
- Accessing Fields in the Result Set
- Cursors
- Getting Column Information
- Stored Procedures
- Getting the Last Insert ID

Establishing a Session

To use SQL Relay, you have to identify the connection that you intend to use.

>     from SQLRelay import PySQLRDB

>     con=PySQLRDB.connect('sqlrserver',9000,'/tmp/test.socket','user','password',0,1)
>     cur=con.cursor()

>     ... execute some queries ...

After calling the connect(), a session is established when the first query is run.

For the duration of the session, the client occupies one of the database connections, so care should be taken to minimize the length of a session.

Executing Queries

Call execute() to run a query.

>     from SQLRelay import PySQLRDB

>     con=PySQLRDB.connect('sqlrserver',9000,'/tmp/test.socket','user','password',0,1)
>     cur=con.cursor()

>     cur.execute('select * from my_table')

>     ... process the result set ...

Commits and Rollbacks

If you need to execute a commit or rollback, you should use the commit() and rollback() methods rather than sending a "commit" or "rollback" query. There are two reasons for this. First, it's much more efficient to call the methods. Second, if you're writing code that can run on transactional or non-transactional databases, some non-transactional databases will throw errors if they receive a "commit" or "rollback" query, but by calling the commit() and rollback() methods you instruct the database connection daemon to call the commit and rollback API methods for that database rather than issuing them as queries. If the API's have no commit or rollback methods, the calls do nothing and the database throws no error. This is especially important when using SQL Relay with ODBC.

Temporary Tables

Some databases support temporary tables. That is, tables which are automatically dropped or truncated when an application closes its connection to the database or when a transaction is committed or rolled back.

For databases which drop or truncate tables when a transaction is committed or rolled back, temporary tables work naturally.

However, for databases which drop or truncate tables when an application closes its connection to the database, there is an issue. Since SQL Relay maintains persistent database connections, when an application disconnects from SQL Relay, the connection between SQL Relay and the database remains, so the database does not know to drop or truncate the table. To remedy this situation, SQL Relay parses each query to see if it created a temporary table, keeps a list of temporary tables and drops (or truncates them) when the application disconnects from SQL Relay. Since each database has slightly different syntax for creating a temporary table, SQL Relay parses each query according to the rules for that database.

In effect, temporary tables should work when an application connects to SQL Relay in the same manner that they would work if the application connected directly to the database.

Catching Errors

If your call to execute() raises an exception, the query failed. You can find out why by catching the exception.

>     from SQLRelay import PySQLRDB

>     con=PySQLRDB.connect('sqlrserver',9000,'/tmp/test.socket','user','password',0,1)
>     cur=con.cursor()

>     try:
>     	cur.execute('select * from my_nonexistant_table')
>     except PySQLRDB.DatabaseError, e:
>     	print e

Bind Variables

Programs rarely execute fixed queries. More often than not, some part of the query is dynamically generated. The Python DB API provides a means for using bind variables in those queries.

For a detailed discussion of binds, see this document.

>     from SQLRelay import PySQLRDB

>     con=PySQLRDB.connect('sqlrserver',9000,'/tmp/test.socket','user','password',0,1)
>     cur=con.cursor()

>     cur.execute('select * from my_table where column1>:val1 and column2=:val2 and column3<:val3',{'val1':1,'val2':'hello','val3':50.546})

>     ... process the result set ...

When passing a floating point number in as a bind or substitution variable, you have to supply precision and scale for the number. See this page for a discussion of precision and scale.

Re-Binding and Re-Execution

A feature of the prepare/bind/execute paradigm is the ability to prepare, bind and execute a query once, then re-bind and re-execute the query over and over without re-preparing it. If your backend database natively supports this paradigm, you can reap a substantial performance improvement.

The Python DB API supports this paradigm via the executemany method. If you pass in a list of parameter dictionaries, the query will be re-executed for each dictionary of bind variable/values.

>     from SQLRelay import PySQLRDB

>     con=PySQLRDB.connect('sqlrserver',9000,'/tmp/test.socket','user','password',0,1)
>     cur=con.cursor()

>     cur.executemany('insert into my_table values (:val1,:val2,:val3)',
>     		[{'val1':1,'val2':'hello','val3':1.11},
>     		{'val1':2,'val2':'hi','val3':2.22},
>     		{'val1':3,'val2':'bye','val3':3,33}])

Accessing Fields in the Result Set

The fetchone(), fetchmany() and fetchall() methods are useful for processing result sets. fetchone() returns a list of values. fetchmany() and fetchall() each return a list of rows where each row is a list of values.

The rowcount member variable gives the number of rows in the result set of a select query or the number of rows affected by an insert/update/delete query.

>     from SQLRelay import PySQLRDB

>     con=PySQLRDB.connect('sqlrserver',9000,'/tmp/test.socket','user','password',0,1)
>     cur=con.cursor()

>     cur.execute('select * from my_table')

>     print 'rowcount:', cur.rowcount

>     print 'the first row:'
>     print cur.fetchone()
>     print

>     print 'the next three rows:'
>     print cur.fetchmany(3)
>     print

>     print 'the rest of the rows:'
>     print cur.fetchall()
>     print

Cursors

Cursors make it possible to execute queries while processing the result set of another query. You can select rows from a table in one query, then iterate through its result set, inserting rows into another table, using only 1 database connection for both operations.

For example:

>     from SQLRelay import PySQLRDB

>     con=PySQLRDB.connect('sqlrserver',9000,'/tmp/test.socket','user','password',0,1)
>     cursor1=con.cursor()
>     cursor2=con.cursor()

>     cursor1.execute('select * from my_huge_table')

>     for a in cursor1.fetchall():
>             cursor2.execute('insert into my_other_table values (:1,:2,:3)',{':1',a[0],':2',a[1],':3',a[2]})

Getting Column Information

After executing a query, column information is stored in the desc variable. desc is a list of tuples. Each tuple corresponds to a column, containing its name, type and length.

>     from SQLRelay import PySQLRDB

>     con=PySQLRDB.connect('sqlrserver',9000,'/tmp/test.socket','user','password',0,1)
>     cur=con.cursor()

>     cur.execute('select * from my_table')

>     for name,type,length in cur.desc:
>             print 'Name:          ', name
>             print 'Type:          ', type
>             print 'Length:        ', length

Stored Procedures

Many databases support stored procedures. Stored procedures are sets of queries and procedural code that are executed inside of the database itself. For example, a stored procedure may select rows from one table, iterate through the result set and, based on the values in each row, insert, update or delete rows in other tables. A client program could do this as well, but a stored procedure is generally more efficient because queries and result sets don't have to be sent back and forth between the client and database. Also, stored procedures are generally stored in the database in a compiled state, while queries may have to be re-parsed and re-compiled each time they are sent.

While many databases support stored procedures. The syntax for creating and executing stored procedures varies greatly between databases.

Stored procedures typically take input paramters from client programs through input bind variables and return values back to client programs either through bind variables or result sets. Stored procedures can be broken down into several categories, based on the values that they return. Some stored procedures don't return any values, some return a single value, some return multiple values and some return entire result sets.

No Values

Some stored procedures don't return any values. Below are examples, illustrating how to create, execute and drop this kind of stored procedure for each database that SQL Relay supports.

Oracle

To create the stored procedure, run a query like the following.

>     create procedure testproc(in1 in number, in2 in number, in3 in varchar2) is
>     begin
>             insert into mytable values (in1,in2,in3);
>     end;

To execute the stored procedure from an SQL Relay program, use code like the following.

>     cur.execute('begin testproc(:in1,:in2,:in3); end;',{'in1':1,'in2':1.1,'in3':'hello'})

To drop the stored procedure, run a query like the following.

>     drop procedure testproc

Sybase and Microsoft SQL Server

To create the stored procedure, run a query like the following.

>     create procedure testproc @in1 int, @in2 float, @in3 varchar(20) as
>             insert into mytable values (@in1,@in2,@in3)

To execute the stored procedure from an SQL Relay program, use code like the following.

>     cur.execute('exec testproc',{'in1':1,'in2':1.1,'in3':'hello'})

To drop the stored procedure, run a query like the following.

>     drop procedure testproc

Firebird

To create the stored procedure, run a query like the following.

>     create procedure testproc(in1 integer, in2 float, in3 varchar(20)) as
>     begin
>             insert into mytable values (in1,in2,in3);
>             suspend;
>     end;

To execute the stored procedure from an SQL Relay program, use code like the following.

>     cur.execute('exec procedure testproc ?, ?, ?',{'in1':1,'in2':1.1,'in3':'hello'})

To drop the stored procedure, run a query like the following.

>     drop procedure testproc

DB2

To create the stored procedure, run a query like the following.

>     create procedure testproc(in in1 int, in in2 double, in in3 varchar(20)) language sql
>     begin
>             insert into mytable values (in1,in2,in3);
>     end;

To execute the stored procedure from an SQL Relay program, use code like the following.

>     cur.execute('call testproc(?,?,?)',{'in1':1,'in2':1.1,'in3':'hello'})

To drop the stored procedure, run a query like the following.

>     drop procedure testproc

Postgresql

To create the stored procedure, run a query like the following.

>     create function testfunc(int,float,varchar(20)) returns void as '
>     begin
>             insert into mytable values ($1,$2,$3);
>             return;
>     end;' language plpgsql

To execute the stored procedure from an SQL Relay program, use code like the following.

>     cur.execute('select testfunc($1,$2,$3)',{'1':1,'2':1.1,'3':'hello'})

To drop the stored procedure, run a query like the following.

>     create function testfunc(int,float,varchar(20))

MySQL

To create the stored procedure, run a query like the following.

>     create procedure testproc(in in1 int, in in2 float, in in3 varchar(20))
>     begin
>             insert into mytable values (in1,in2,in3);
>     end;

To execute the stored procedure from an SQL Relay program, use code like the following.

>     cur.execute('call testproc(?,?,?)',{'1':1,'2':1.1,'3':'hello'})

Note: Versions of MySQL prior to 5.0 had trouble calling stored procedures using bind variables. If you are using a version of MySQL prior to 5.0 then SQL relay must fake the bind variables and you must use colon-delimited variables (:1, :2, :3, etc.) in your queries rather than the native-mysql queston marks.

To drop the stored procedure, run a query like the following.

>     drop procedure testproc

Single Values

Some stored procedures return single values. Below are examples, illustrating how to create, execute and drop this kind of stored procedure for each database that SQL Relay supports.

Oracle

To create the stored procedure, run a query like the following.

>     create function testproc(in1 in number, in2 in number, in3 in varchar2) returns number is
>     begin
>             return in1;
>     end;

To execute the stored procedure from an SQL Relay program, use code like the following.

>     cur.execute('select testproc(:in1,:in2,:in3) from dual',{'in1':1,'in2':1.1,'in3':'hello'})
>     result=cur.fetchone()[0]

To drop the stored procedure, run a query like the following.

>     drop function testproc

Sybase and Microsoft SQL Server

In Sybase and Microsoft SQL Server, stored procedures return values through output parameters rather than as return values of the procedure itself. However, the SQL Relay Python DBdriver does not currently support output parameters.

Firebird

To create the stored procedure, run a query like the following.

>     create procedure testproc(in1 integer, in2 float, in3 varchar(20)) returns (out1 integer) as
>     begin
>             out1=in1;
>             suspend;
>     end;

To execute the stored procedure from an SQL Relay program, use code like the following.

>     cur.execute('select * from testproc(?,?,?)',{'in1':1,'in2':1.1,'in3':'hello'})
>     result=cur.fetchone()[0]

To drop the stored procedure, run a query like the following.

>     drop procedure testproc

DB2

In DB2, stored procedures return values through output parameters rather than as return values of the procedure itself. However, the SQL Relay Python DB driver does not currently support output parameters.

Postgresql

To create the stored procedure, run a query like the following.

>     create function testfunc(int,float,char(20)) returns int as '
>     declare
>             in1 int;
>             in2 float;
>             in3 char(20);
>     begin
>             in1:=$1;
>             return;
>     end;
>     ' language plpgsql

To execute the stored procedure from an SQL Relay program, use code like the following.

>     cur.execute('select * from testfunc($1,$2,$3)',{'1':1,'2':1.1,'3':'hello'})
>     result=cur.fetchone()[0]

To drop the stored procedure, run a query like the following.

>     drop function testfunc(int,float,char(20))

MySQL

A single value can be returned from a MySQL function.

To create the function, run a query like the following.

>     create function testfunc(in in1 int, in in2 float, in in3 varchar(20)) returns int return in1;

To execute the function from an SQL Relay program, use code like the following.

>     cur.execute('select testfunc(?,?,?)',{'in1':1,'in2':1.1,'in3':'hello'})

Note: Versions of MySQL prior to 5.0 had trouble calling stored procedures using bind variables. If you are using a version of MySQL prior to 5.0 then SQL relay must fake the bind variables and you must use colon-delimited variables (:1, :2, :3, etc.) in your queries rather than the native-mysql queston marks.

To drop the function, run a query like the following.

>     drop function testfunc

A single value can be returned in the result set of a MySQL procedure.

To create the procedure, run a query like the following.

>     create procedure testproc() begin select 1; end;

To execeute the procedure from an SQL Relay program, use code like the following.

>     cur.execute('select testfunc()')
>     result=cur.fetchone()[0]

To drop the procedure, run a query like the following.

>     drop procedure testproc

A single value can be returned using the output variable of a MySQL procedure.

To create the procedure, run a query like the following.

>     create procedure testproc(out out1 int) begin select 1 into out1; end;

To execeute the procedure from an SQL Relay program, use code like the following.

>     cur.execute('set @out1=0')
>     cur.execute('call testproc()')
>     cur.execute('select @out1')
>     result=cur.fetchone()[0]

To drop the procedure, run a query like the following.

>     drop procedure testproc

Multiple Values

Some stored procedures return multiple values. Below are examples, illustrating how to create, execute and drop this kind of stored procedure for each database that SQL Relay supports.

Oracle

In Oracle, stored procedures can return values through output parameters or as return values of the procedure itself. If a procedure needs to return multiple values, it can return one of them as the return value of the procedure itself, but the rest must be returned through output parameters. However, the SQL Relay Python DB driver does not currently support output parameters.

Sybase and Microsoft SQL Server

In Sybase and Microsoft SQL Server, stored procedures return values through output parameters rather than as return values of the procedure itself. However, the SQL Relay Python DB driver does not currently support output parameters.

Firebird

To create the stored procedure, run a query like the following.

>     create procedure testproc(in1 integer, in2 float, in3 varchar(20)) returns (out1 integer, out2 float, out3 varchar(20)) as
>     begin
>             out1=in1;
>             out2=in2;
>             out3=in3;
>             suspend;
>     end;

To execute the stored procedure from an SQL Relay program, use code like the following.

>     cur.execute('select * from testfunc(?,?,?)',{'in1':1,'in2':1.1,'in3':'hello'})
>     out1=cur.fetchone()[0]
>     out2=cur.fetchone()[1]
>     out3=cur.fetchone()[2]

To drop the stored procedure, run a query like the following.

>     drop procedure testproc

DB2

In DB2, stored procedures return values through output parameters rather than as return values of the procedure itself. However, the SQL Relay Python DB driver does not currently support output parameters.

Postgresql

To create the stored procedure, run a query like the following.

>     create function testfunc(int,float,char(20)) returns record as '
>     declare
>             output record;
>     begin
>             select $1,$2,$3 into output;
>             return output;
>     end;
>     ' language plpgsql

To execute the stored procedure from an SQL Relay program, use code like the following.

>     cur.execute('select * from testfunc($1,$2,$3) as (col1 int, col2 float, col3 char(20))',{'1':1,'2':1.1,'3':'hello'})
>     out1=cur.fetchone()[0]
>     out2=cur.fetchone()[1]
>     out3=cur.fetchone()[2]

To drop the stored procedure, run a query like the following.

>     drop function testfunc(int,float,char(20))

MySQL

Here's how you can get multiple values from the result set of a MySQL procedure.

To create the stored procedure, run a query like the following.

>     create procedure testproc(in in1 int, in in2 float, in in3 varchar(20)) begin select in1, in2, in3; end;

To execute the stored procedure from an SQL Relay program, use code like the following.

>     cur.execute('select testfunc(?,?,?)',{'1':1,'2':1.1,'3':'hello'})
>     out1=cur.fetchone()[0]
>     out2=cur.fetchone()[1]
>     out3=cur.fetchone()[2]

Note: Versions of MySQL prior to 5.0 had trouble calling stored procedures using bind variables. If you are using a version of MySQL prior to 5.0 then SQL relay must fake the bind variables and you must use colon-delimited variables (:1, :2, :3, etc.) in your queries rather than the native-mysql queston marks.

To drop the stored procedure, run a query like the following.

>     drop procedure testproc

Here's how you can get multiple values from the output variables of a MySQL procedure.

To create the stored procedure, run a query like the following.

>     create procedure testproc(out out1 int, out out2 float, out out3 varchar(20)) begin select 1,1.1,'hello' into out1, out2, out3; end;

To execute the stored procedure from an SQL Relay program, use code like the following.

>     cur.execute('set @out1=0, @out2=0.0, @out3=\'\'')
>     cur.execute('call testfunc(@out1,@out2,@out3)')
>     cur.execute('select @out, @out2, @out3')
>     out1=cur.fetchone()[0]
>     out2=cur.fetchone()[1]
>     out3=cur.fetchone()[2]

To drop the stored procedure, run a query like the following.

>     drop procedure testproc

Result Sets

Some stored procedures return entire result sets. Below are examples, illustrating how to create, execute and drop this kind of stored procedure for each database that SQL Relay supports.

Oracle

Stored procedures in Oracle can return open cursors as return values or output parameters. A client-side cursor can be bound to this open cursor and rows can be fetched from it. However, the SQL Relay Python DB driver does not currently support output parameters.

Sybase and Microsoft SQL Server

To create the stored procedure, run a query like the following.

>     create procedure testproc as select * from testtable

To exceute the stored procedure from an SQL Relay program, ue code like the following.

>     cur.execute('call testproc')
>     result=cur.fetchall()

To drop the stored procedure, run a query like the following.

>     drop procedure testproc

Firebird

Stored procedures in Firebird can return a result set if a select query in the procedure selects values into the output parameters and then issues a suspend command, however SQL Relay doesn't currently support stored procedures that return result sets.

DB2

Stored procedures in DB2 can return a result set if the procedure is declared to return one, however SQL Relay doesn't currently support stored procedures that return result sets.

Postgresql

To create the stored procedure, run a query like the following.

>     create function testfunc() returns setof record as '
>             declare output record;
>     begin
>             for output in select * from mytable loop
>                     return next output;
>             end loop;
>             return;
>     end;
>     ' language plpgsql

To execute the stored procedure from an SQL Relay program, use code like the following.

>     cur.execute('select * from testfunc() as (col1 int, col2 float, col3 char(20))',{'in1':1,'in2':1.1,'in3':'hello'})
>     result=cur.fetchall()

To drop the stored procedure, run a query like the following.

>     drop function testfunc

MySQL

The result sets of all select statements called within MySQL stored procedures (that aren't selected into variables) are returned from the procedure call. Though MySQL stored procedures can return multiple result sets, currently SQL Relay can only fetch the first result set.

To create the stored procedure which returns a result set, run a query like the following.

>     create procedure testproc() begin select * from mytable; end;

To execute the stored procedure from an SQL Relay program, use code like the following.

>     cur.execute('call testfunc()',{'1':1,'2':1.1,'3':'hello'})
>     result=cur.fetchall()

To drop the stored procedure, run a query like the following.

>     drop procedure testproc

Getting the Last Insert ID

Databases with autoincrement or identity columns often provide functions which return the "last insert id"; the value of the autoincrement column that was generated during the insert into the database.

Unfortunately Python DB doesn't expose a generic method for getting the last insert id, but when using the SQLite database, you can get the last insert id by running the query:

> select last insert rowid