I’ve moved

I am currently blogging from my new website which is at https://curiousdba.netlify.com/. I will see you there!

 

Peoplesoft performance monitor

EDIT: I have an updated version of this page including some more information on my new blog.

I keep having to look up how this works, so thought I would make a note here.

 

Systems

First the systems are kept in sysadm.pspmsysdefn.

SQL> desc sysadm.pspmsysdefn
 Name                           Null?    Type
 ------------------------------ -------- ----------------------
 PM_SYSTEMID                    NOT NULL NUMBER(38)
 DBNAME                         NOT NULL VARCHAR2(8 CHAR)
 DESCR60                        NOT NULL VARCHAR2(60 CHAR)
 PM_BUFF_INT                    NOT NULL NUMBER(38)
 PM_MAX_BUFF                    NOT NULL NUMBER(38)
 PM_SAMPLE_INT                  NOT NULL NUMBER(38)
 PM_PING_INT                    NOT NULL NUMBER(38)
 PM_MAX_HIST_AGE                NOT NULL NUMBER(38)
 PM_ARCHIVE_MODE                NOT NULL VARCHAR2(1 CHAR)
 PM_MAX_TRANS_TMOUT             NOT NULL NUMBER(38)
 GUID                           NOT NULL VARCHAR2(36 CHAR)
 PM_USER_TRACE                  NOT NULL VARCHAR2(1 CHAR)
 PM_SAMPLING_RATE               NOT NULL NUMBER(38)
 PM_FILTER_LEVEL3               NOT NULL VARCHAR2(2 CHAR)
 LASTUPDOPRID                   NOT NULL VARCHAR2(30 CHAR)
 LASTUPDDTTM                             TIMESTAMP(6)

Interesting columns are PM_SYSTEMID, which we will need later. DBNAME seems to come from the dbname in ps.psdbowner in the monitored system. The GUID comes from SYSADM.PSOPTIONS.GUID. It can be useful to reset this if you want a refreshed system to show up as the same in the monitor.

Agents

The next table is the agents. These are defined in SYSADM.PSPMAGENT.

SQL> desc sysadm.pspmagent
 Name                          Null?    Type
 ----------------------------- -------- ----------------------------
 PM_AGENTID                    NOT NULL NUMBER(10)
 PM_AGENT_TYPE                 NOT NULL VARCHAR2(32 CHAR)
 PM_DOMAIN_NAME                NOT NULL VARCHAR2(32 CHAR)
 PM_DOMAIN_TYPE                NOT NULL VARCHAR2(2 CHAR)
 PM_DOMAIN_DIR                 NOT NULL VARCHAR2(254 CHAR)
 PM_DOMAIN_MONITOR             NOT NULL VARCHAR2(1 CHAR)
 PM_HOST_PORT                  NOT NULL VARCHAR2(64 CHAR)
 PM_INSTANCE                   NOT NULL VARCHAR2(32 CHAR)
 PM_SYSTEMID                   NOT NULL NUMBER(38)
 PM_AGENT_INACTIVE             NOT NULL VARCHAR2(1 CHAR)

Important columns here are pm_systemid from the table above, and the agentid. The rest of the columns describe the agent. There is one agent per website, then one per domain, one per process in the apps server and process scheduler. In a production sized system I am looking at there are 100 agents.

To look at the agents from system 5, I used:

select * from sysadm.pspmagent where pm_systemid = 5;

The records being collected go to sysadm.pspmranscurr. Once they have been collected they are moved to pspmtranshist.

 Name                 Null?    Type
 -------------------- -------- ----------------------------
 PM_INSTANCE_ID       NOT NULL NUMBER(20)
 PM_TRANS_DEFN_SET    NOT NULL NUMBER(38)
 PM_TRANS_DEFN_ID     NOT NULL NUMBER(38)
 PM_AGENTID           NOT NULL NUMBER(10)
 PM_TRANS_STATUS      NOT NULL VARCHAR2(1 CHAR)
 OPRID                NOT NULL VARCHAR2(30 CHAR)
 PM_PERF_TRACE        NOT NULL VARCHAR2(30 CHAR)
 PM_CONTEXT_VALUE1    NOT NULL VARCHAR2(254 CHAR)
 PM_CONTEXT_VALUE2    NOT NULL VARCHAR2(254 CHAR)
 PM_CONTEXT_VALUE3    NOT NULL VARCHAR2(254 CHAR)
 PM_CONTEXTID_1       NOT NULL NUMBER(38)
 PM_CONTEXTID_2       NOT NULL NUMBER(38)
 PM_CONTEXTID_3       NOT NULL NUMBER(38)
 PM_PROCESS_ID        NOT NULL NUMBER(38)
 PM_AGENT_STRT_DTTM            TIMESTAMP(6)
 PM_MON_STRT_DTTM              TIMESTAMP(6)
 PM_TRANS_DURATION    NOT NULL NUMBER(21,3)
 PM_PARENT_INST_ID    NOT NULL NUMBER(20)
 PM_TOP_INST_ID       NOT NULL NUMBER(20)
 PM_METRIC_VALUE1     NOT NULL NUMBER(28,4)
 PM_METRIC_VALUE2     NOT NULL NUMBER(28,4)
 PM_METRIC_VALUE3     NOT NULL NUMBER(28,4)
 PM_METRIC_VALUE4     NOT NULL NUMBER(28,4)
 PM_METRIC_VALUE5     NOT NULL NUMBER(28,4)
 PM_METRIC_VALUE6     NOT NULL NUMBER(28,4)
 PM_METRIC_VALUE7     NOT NULL VARCHAR2(128 CHAR)
 PM_ADDTNL_DESCR               CLOB

Most of these columns are interesting. Think of a transaction moving from the web server to the application server then to the database and back. At each point the time taken is recorded.

trans_defn_set is always 1 on my system, which means peopletools.

pm_trans_defn_id is a 3 digit number which gives an idea of how far into the stack the metric is being generated. So anything in the 100 range is in the web server, and anything in the 400 range is in the application server.

pm_agent_id can be translated to an agent name using the aforementioned sysadm.pspmagent. This also gives a lot of information as to where the wait is occurring.

pm_context_id 1-3 are the labels for the relevant pm_context_value fields. They are translated by the table sysadm.pspmcontextdefn.

pm_metric_id 1-7 performs a similar task for the pm_metric_value fields, and are translated using sysadm.pspmmetricdefn.

We can also see that some of the instances are related to each other by pm_instance_id, pm_parent_instance_id and pm_top_instance_id. We can use analytic functions to display these nicely as follows:

select ind, oprid, montime, dur, typ, pm_trans_label, hostport,
 label1, value1, label2, value2, value7, clabel1, cvalue1, clabel2,
 cvalue2, clabel3, cvalue3, descr from (
  select LPAD('*', 2*level-1,'-')||to_char(level) as ind, x.* from (
   select ta.oprid as oprid, 
    to_char(ta.PM_MON_STRT_DTTM,'DD/MM/yy hh24:mi:ss') as montime,
    ta.PM_TRANS_DURATION as dur, ta.pm_trans_defn_id as typ, 
    td.PM_TRANS_LABEL, ta.pm_top_inst_id as topid,
    ta.pm_instance_id as pm_instance_id,
    ta.PM_PARENT_INST_ID as PM_PARENT_INST_ID,
    a.pm_host_port as hostport, md1.PM_METRICLABEL as label1,
    ta.PM_METRIC_VALUE1 as value1, md2.PM_METRICLABEL as label2,
    pm_metric_value2 as value2, md7.PM_METRICLABEL as label7,
    pm_metric_value7 as value7, cd1.pm_context_label as clabel1,
    ta.PM_CONTEXT_VALUE1 as cvalue1, cd2.pm_context_label as clabel2,
    ta.PM_CONTEXT_VALUE2 as cvalue2, cd3.pm_context_label as clabel3, 
    ta.PM_CONTEXT_VALUE3 as cvalue3, ta.PM_ADDTNL_DESCR as descr
   from sysadm.pspmtranshist ta, sysadm.PSPMAGENT a,
    sysadm.pspmcontextdefn cd1, sysadm.pspmcontextdefn cd2,
    sysadm.pspmcontextdefn cd3, sysadm.pspmtransdefn td, 
    sysadm.PSPMMETRICDEFN md1, sysadm.PSPMMETRICDEFN md2,
    sysadm.PSPMMETRICDEFN md7
   where 
    a.PM_AGENTID = ta.PM_AGENTID
    and cd1.pm_contextid = ta.pm_contextid_1
    and cd2.pm_contextid = ta.pm_contextid_2
    and cd3.pm_contextid = ta.pm_contextid_3
    and ta.pm_top_inst_id in (
     select distinct pm_top_inst_id 
     from sysadm.pspmtranshist 
     where PM_MON_STRT_DTTM >
           to_timestamp('02-06-2016 16:00','dd-mm-yyyy hh24:mi') 
     and PM_MON_STRT_DTTM <
           to_timestamp('02-06-2016 17:00','dd-mm-yyyy hh24:mi') 
     and pm_trans_duration > 10000)
    and td.pm_trans_defn_id = ta.pm_trans_defn_id
    and md1.pm_metricid = td.pm_metricid_1
    and md2.pm_metricid = td.pm_metricid_2
    and md7.pm_metricid = td.pm_metricid_7
   ) x
  start with x.pm_parent_inst_id = 0
  connect by prior x.pm_instance_id = x.pm_parent_inst_id
);

 

Setting up lineage in linux for OBIEE

We have Oracle Business Intelligence Enterprise Edition running on Linux. I was asked to install lineage, but while it is supported to run on Linux, it appears the install is only supported on Windows. Of course we are not licensed to run it on Windows and Oracle would not allow us to install and copy across – you need to be licensed.

However with the help of someone from Oracle I found that you can install it on linux using the following procedure.

Prepare the companion CD

Download the ODI companion CD from oracle edelivery, and unzip the whole thing as an oracle home. Go to misc/biee-lineage and unzip odiobilineage.zip

For version 11.1.1.7 we found we had to apply patch 17008493. Follow the instructions in the readme, but install OPatch in the home, and apply the patch using:

$ opatch apply -jre /path/to/jre

Merge the rpd file

The RPD needs to be merged with the lineage rpd. This is done by converting it to UDML and uploading. Create a temporary directory and copy in odi_repository_archive_11g.rpd from the ODI companion CD under misc/biee-lineage/artifacts/11g. Then run:

ORACLE_HOME=/path/to/middleware_home
. $ORACLE_HOME/instances/$inst/bifoundation/OracleBIApplication/coreapplication/setup/bi-init.sh
$ORACLE_HOME/Oracle_BI1/bifoundation/server/bin/nqudmlgen \
       -P Admin123 \
       -R ./odi_repository_archive_11g.rpd -O lineage.udml \
       -N -Q

Now copy in the RPD file and merge in the lineage UDML using:

 $ORACLE_HOME/Oracle_BI1/bifoundation/server/bin/nqudmlexec\
       -P Password \
       -I lineage.udml \
       -B original.rpd \
       -O updated.rpd

---------------lineage.udml---------------

      ---------------Complete Success!!---------------

Where Password is the password of the rpd file. Copy the RPD file to windows, and update the connection pool as per Oracles lineage documentation.

Find the ORACLE_ODI_REPOSITORY. Edit the connection pool to match the ODI Work repository.

• Expand the ORACLE_ODI_REPOSITORY database in the OBIEE Physical Layer, double-click the Connection Pool node, and edit the Connection Pool to match your ODI work repository configuration:
  • Update the Data source name, Username and Password fields.
  • Click OK.
  • Right-click the Physical schema and rename it to match the schema of the ODI Work Repository.
  • Click OK to save your changes.
• Expand the renamed schema and test this updated connection as follows:
  • Right-click one of the tables of this physical schema and updating the row count.
  • Right-click the same table again and select View data to view data with the updated row count.

This completes thework with the RPD file. It can now be uploaded back to the server.

Copy the resources

Now we need to copy the resources from the CD to the correct location on the server. Under the CD change to misc/biee-lineage/artifacts/images and copy the two images to $ORACLE_HOME/user_projects/domains/bifoundation_domain/servers/bi_server1/tmp/_WL_user/analytics_11.1.1/7dezjl/war/res

Web catalog

I did it manually. The manual process needs to be run on  the server because the client version had missing libraries that are required.

Make sure that X is working (Run xclock to check) and run:

cd $ORACLE_HOME/instances/$instance/bifoundation/OracleBIPresentationServicesComponent/coreapplication_obips1/catalogmanager
./runcat.sh

This brings up the catalogue manager. Select “Open Catalog” from the File menu. Select offline, then browse to the catalogue location which is:

$ORACLE_HOME/instances/$INSTANCE/bifoundation/OracleBIPresentationServicesComponent/coreapplication_obips1/catalog/SampleAppLite

You should see your developers directories under here. Using the online catalogue also works. Click OK.

Extract the web catalog using file->Unarchive and select

$ODI_COMPANION/misc/biee-lineage/artifacts/11g/odi_catalog_archive_11g.cat

Export the web catalog report using Tools->Create Report.

The report type should be analysis. Add the following fields to the report in the following order: “Owner”, “Folder”, “Name” ,”Subject Area” ,”Formula” ,”Table”, “Column”

Save the report

It should be possible to script this step as follows:

OBIEE_WEBCAT=SampleAppLite
WIN_EXP_RPD_DIR="D:\tmp"
OBIEE_WEBCAT_MGR=$OBIEE_INSTANCE_HOME/instances/instance1/bifoundation/OracleBIPresentationServicesComponent/coreapplication_obips1/catalogmanager
OBIEE_WEBCAT_BASE=$OBIEE_INSTANCE_HOME/instances/instance1/bifoundation/OracleBIPresentationServicesComponent/coreapplication_obips1/catalog

$OBIEE_WEBCAT_MGR/runcat.sh -cmd report \
 -offline $OBIEE_WEBCAT_BASE/$OBIEE_WEBCAT \
 -forceOutputFile $ODL_TMP_DIR/webcat_doc.txt \
 -distinct \
 -folder /shared \
 -type Analysis "Owner" "Folder" "Name" "Subject Area" "Formula" "Table" "Column"bb

Export the RPD report

Open the RPD file in the OBIEE administration tool. Select Tools->Utilities. Select Repository Documentation and click execute. Save the file.

Run lineage

Gather the webcat extract file, and the RPD report into a directory. Create a file to refresh lineage that looks like this:

#Properties for ODI/OBIEE Lineage
#Fri Dec 13 09:45:15 EST 2013
ODI_MASTER_USER=ODI_REPO_M
ODI_MASTER_PASS=firtr33s
ODI_MASTER_DRIVER=oracle.jdbc.OracleDriver
ODI_MASTER_URL=jdbc\:oracle\:thin\:@palin\:1532\:BIPROD
ODI_SUPERVISOR_USER=SUPERVISOR
ODI_SUPERVISOR_PASS=firtr33s
ODI_SECU_WORK_REP=WORKREPDEV
OBIEE_VERSION=11g
OBIEE_WEBCAT_EXPORT_FILE=/path/to/webcat.txt
OBIEE_RPD_EXPORT_FILE=/path/to/repo.txt
MAPPING_FILE=/path/to/MappingData.csv
INSTALL_ODI_LINEAGE=no
EXPORT_OBIEE_METADATA=no

Create the mapping data file. The layout for these is on the companion CD at:

misc/biee-lineage/samples/MappingData.csv

The format is:

ORACLE_SID,DB_USERID,,nnnn

Where:

ORACLE_SID is the oracle_sid of the database (Well the TNS connection)

DB_USERID is  the username within the database (Probably DEV_BIPLATFORM)

nnnn is the ODI Model ID. This can be derived by opening the ODI tool and connecting to the master and work repository. Select the designer tab on the left, and open the model. The model id is under version.

Now it should be possible to refresh lineage. The script needs to be run from the bin directory as it has relative paths. Also JAVA_HOME needs to be set.

JAVA_HOME=/usr/java
cd $odi_companion/misc/biee-lineage/bin
./refreshlineage.sh -propertyFile=$LINDIR/lineage.properties \
                    -mappingFile=$LINDIR/MappingData.csv

All being well, lineage will be refreshed and everything will work. Presumably the catalogue extract and rpd file extract will need to be updated periodically as they change.

Health check of an Oracle system

I like to do a healthcheck of my Oracle system on a quarterly basis. This is because the report is hard to generate, and it is the longest timespan I thought I could get away with Monthly would be better, but until I can automate the reports generation this would take too much time.

The reason for the report is to keep management informed about what we are doing. To ensure the system is healthy and to ensure we are applying best practices in maintaining the system. It also serves as a mechanism to alert management (and myself) to potential problems.

Management will not have time to read the report, so it is important to have a summary at the top giving an overview of the most important things that have happened over the last quarter, tasks in progress, risks and any other things you want them to see.

The full healthcheck should contain the following:

Service improvements – An overview of work done since the last report.

Patch levels – Have the latest critical patches been applied to all tiers, and if not why not.

Software levels. Is all software in premier support, and if not why not. When does premier support end, and what plans are there to upgrade.

Hardware – For each hardware tier, is the hardware sufficient to cope with  the application. Break this down into CPU, I/O, disc space, Memory usage, and network usage. Graphs are good here to show trends during the day, week, month year and over time.

Performance – Ideally include some measure of user visible performance. Again graphs showing trends are useful here.

For the database attach an AWR report or similar showing top SQL for each metric. I also include the top hot objects and the SQL updating them. We found a number of tuning targets this way. Database I/O has caused problems for us in the past, so I graph redo generation. I create a graph of the growth of the top 10 application tables, system tables, lobs and indexes. Again over time this can point to trends and help ask the business whether the audit data from 8 years ago is really required. I point to documentation for non standard parameters and why they are set.

Also take the opportunity to compare against some of Oracles health check/recommended practices documents. There are likely to be several to choose from, for security, recommended performance configuration etc. Document any deviations from the document and the reason for it. Equally be wary of changing  the application to match Oracles recommendation without thorough testing, or understanding the reason for the changes.

I find the health check is a very valuable process for everyone concerned. The document is a little tedious to write, but automating its creation too much would reduce its effectiveness.

Upgrading tuxedo

Applying a patch to tuxedo I got:

 

"Must specify the correct TUXDIR"

 

I think the patch installed, but I like to have a success message. I ran the uninstall and investigated.

It happened because the tuxedo directory was copied from another server. Oracle has set up a mini registry which the upgrade tries to edit, and if it can not, this error appears. The following files need to be copied:

When tuxedo is installed, you create an ORACLE_HOME directory to contain this registry information. In my installation, this was the parent directory of the TUXDIR. It contains a registrry.xml file and a log directory. These should also be copied from the same install as the TUXDIR was copied from, then edit the registry.xml file to update the host name.

 

Then install the patch again:

$ ./install
DIR=/psoft/bea_patches/11g_90/RP090
rpreleasenote =  SUSE LINUX Enterprise Server 10 (x86_64) x86 for AMD64, 64bits Tuxedo
portreleasenotes= SUSE LINUX Enterprise Server 10 (x86_64) x86 for AMD64, 64bits Tuxedo
Installing server and client files...
Enter owner for patch files:
psoft
Enter group for patch files:
psoft

The patch installation finished successfully.

Hooray!

Raspberry Pi Python introduction part 3 – Sharing nicely

When I run the last program on my laptop, pretty soon the fan starts whirring and it gets hot. On the raspberry pi the program uses all the CPU it can get, the CPU graph goes completely green. Another problem is that it will run differently on different CPUs. On a fast CPU it will run faster and on a slow CPU it will run slower. This is not a real problem for this program, but if it was a game it would be really annoying to play at 10 times the proper speed.

We should share the CPU nicely with other programs, and keep the program running at the correct speed. Pygame gives us a way to do this, using the clock object. Lets write another program to demonstrate this, and we can add a little interaction as well.

Looking at the program we wrote last time, we can strip away the parts that are special for that program and end up with a skeleton which we can build on for future projects. It looks like this:

import pygame, random, sys

# You have to call this to make it work.
pygame.init()

# Set up some variables containing the screeen size
sizeX=600
sizeY=600

# Start drawing from the middle of the window.
x=sizeX/2
y=sizeY/2

# Set the starting colour
colour = pygame.Color('#444444')

# Create the pygame window
window = pygame.display.set_mode([sizeX,sizeY])


# Create the clock object
clock = pygame.time.Clock()


# Put a name on the window.
pygame.display.set_caption("Random art")

# This will loop forever.
while True:
    #-- Program loop goes here.

    # Wait for the next tick.
    clock.tick(40)

    pygame.display.flip()
    for event in pygame.event.get():
        if event.type == pygame.QUIT:
            sys.exit()

I have added in the clock stuff. Just two lines. The first creates the clock object, and the second one makes the program wait. It makes the loop run no more than 40 times a second, which is plenty. This means that we shouldn’t make the loop run for more than 1/40th of a second. This seems like a terriby short amount of time, but processors are really quick these days. Even the raspberry pi with its relatively slow CPU runs about 1,000,000,000 (One thousand million) instructions per second, which, if we divide that by 40 gives us 25,000,000 (twenty five million instructions) for each time the loop runs. Of course this is talking about a machine code instruction, a python instruction takes a lot of machine code instructions. Even so, we will not need this many!

This time I decided to make a spirograph. Wikipedia shows us the equations we need. Don’t worry, I have typed them in below. We need a counter, which we should initialise to zero at the start of the loop. We will also create random values for k and l, which should be between 0 and 1. These determine the shape of the spirograph pattern. We should pick a value for R which makes the spirograph fill the screen.

k=random.random()
l=random.random()
i=0
R=600

random.random() gives a random number between 0 and 1, which is exactly what we want.

The equations which need to go in the loop are taken from wikipedia:

    newx = R*( (1-k)*math.cos(t) + l*k*math.cos((1-k)*t/k) )
    newy = R*( (1-k)*math.sin(t) - l*k*math.sin((1-k)*t/k) )

Note that we are using the math module, so we need to add that to the import line at the start. When we look at the documentation for math, we see that math.sin and math.cos take an angle in radians. There are pi (3.14) radians in a circle, but 360 degrees. It would look better if we counted in degrees. The math module can convert degrees to radians for us. If we use i for the counter, we can create t as follows:

    t = math.radians(i)

This goes above the lines that calculate x and y above.

Then we need to draw the line, and set up x and y as before. We should set x and y to 0 at the start of the loop and not draw if they are both zero. This means the first time a line is drawn is on the second time through the loop, when we have two points to draw a line between. Otherwise we will get a line from the top left going to the start which will look ugly.

    if x==0 and y==0:
       pass
    else:
       pygame.draw.line(window,colour,(x,y),(newx,newy),2)
    x=newx
    y=newy

The double equals sign means check if it is equal to. The single equal sign means assign. These are two separate things and it is easy to get them confused. Fortunately python will not allow an assignment in an if statement, so it reminds you if you forget (Try it!)

And of course we must not forget to increment our counter.

   i=i+5

The more you increment i by, the faster the line will draw, but if you increment it by too much it will start to look jagged.

If you managed to keep up, try running the program now. You will get a quarter of a spirograph like the picture on the right. This is because our window only shows values where x and y are both greater than zero.

To fix this is actually quite easy. First we should shrink the image so it can fit in the window. Change R from 600 to 300. Then in the draw line we just need to add 300 to each value like this:

       pygame.draw.line(window,colour,(x+R,y+R),(newx+R,newy+R),2)

This looks much better. A whole spirograph.

Next time it will be time to start adding some interaction into our program. It is more fun if you can make it do stuff!

That was quite complicated. In case you couldn’t follow everything above, here is the whole program.

Let me know if you think of a way to make my explanation clearer. Also there must be a way to make the lines eventually join up like a real spirograph. If you know what it is, leave a comment!

import pygame, random, sys, math

# You have to call this to make it work.
pygame.init()

# Set up some variables containing the screeen size
sizeX=600
sizeY=600

# Set the starting colour
colour = pygame.Color('#009900')

# Create the pygame window
window = pygame.display.set_mode([sizeX,sizeY])

# Create the clock object
clock = pygame.time.Clock()

# Put a name on the window.
pygame.display.set_caption("Spirograph")

# Initialise more variables
# k and l are numbers between 0 and 1.
# k is the ratio of the distance of the small circle from the big circle
l=random.random()

# l is the ratio of the small circles radius (to the hole with the pen in)
# to the distance from the centre of the large circle.
k=random.random()

# Counter - real number - not integer.
i=0.0

# Scaling factor (Radius of big circle)
R=300

x=0
y=0

# This will loop forever.
while True:
    t = math.radians(i)

    newx = R * ((1-k) * math.cos(t) + l*k*math.cos((1-k) * t / k ))
    newy = R * ((1-k) * math.sin(t) - l*k*math.sin((1-k) * t / k ))

    if (x==0 and y==0):
       pass
    else:
       pygame.draw.line(window,colour,(x+R,y+R),(newx+R,newy+R),2)
    x=newx
    y=newy

    i=i+5

    clock.tick(40)
    pygame.display.flip()
    for event in pygame.event.get():
        if event.type == pygame.QUIT:
            sys.exit()
        elif event.type == pygame.KEYDOWN:
            if event.key == pygame.K_ESCAPE:
                sys.exit()
sys.exit()

Raspberry pi – The politics

I was wondering why has there not been a computer like the Raspberry Pi till now. The technology was there, so why didn’t someone put it together till now?

Companies exist to make money. They sell smart phones for £500. They do not sell computers for £30. How do you make money like that?

So it took some inspired public minded people to create the computer. But what about the software? You can’t install Microsoft Windows 7 on an ARM chip. Anyway, how much would Microsoft be able to charge for it? What they needed is an operating system that already worked on an ARM chip. Ideally one that was being given away for free. Fortunately there are some inspired public minded people called Debian who make a complete operating system, together with the programs you can run on it.

At the moment, the two have only just come together. This means there are some gaps where they join. Some work still need to be done to get them working well together. People will do this work and give it away. Over the next few months

Debian only exists because of other groups of public minded people wrote software and gave it away for free. In itself, that would not be useful, because programs that run on desktops will not work on ARM chips. You need the source code.

In the 1980s a man called Richard Stallman created an organisation called the Free Software Foundation. Their goal was to create a complete operating system, and give it away complete with the source. This was almost done. They even had a name for it – GNU, but they were having one problem. It was the kernel, the central part of the operating system which talks to the hardware. Amazingly another public minded individual had created something of a group around a kernel he was writing. He was Linus Torvalds, and the kernel was called Linux.

Computers were getting graphical interfaces around that time. Fortunately the Massachusetts Institute of Technology (MIT) was giving away its X windows system.

So we have the Raspberry Pi foundation providing the computer. Linus Torvalds and his group provided the kernel. The free software foundation provided the GNU operating system, MIT providing the windows system, thousands of other people writing programs, and Debian putting them all together.

We owe these people a debt of gratitude for what they have done for us, particularly Richard Stallman for his vision to write software, and give it away under the condition that anyone who receives it and passes it on, also passes on all the rights they got. Some people call it share and share alike.

I hope this inspires you to give something back. When you write programs, put them under a free software license. Consider the GNU Affero General Public License – but read it first, so you see what it means.

Its arrived – how to get started

Hooray! My raspberry pi arrived. I plugged it in, powered it up, and it didn’t work. It seems that the power supply was not powerful enough. So I got a more powerful one, and it worked.

Here is what I did. I downloaded the debian image from the raspberry pi downloads page, and wrote it to an SD card as suggested on the downloads page.

The first time I booted the system it did some configuration and seemed to hang. The messages said it was OK to reboot if it hung so I did. Then it booted to a text prompt.


Debian GNU/Linux 6.0 raspberrypi tty1

raspberrypi login:


I logged in following the instructions on the download page. It is a standard bash prompt which is really simple to use if you know the commands. Maybe that is a tutorial for another day. This time all I did was type:


startx


This brings up a user interface like computers used to have in 1999. This computer is probably about as powerful as a computer of that era though it uses much less electricity and can drive a 1080p monitor.

The first thing that strikes you is the picture of the raspberry pi logo on the background. Apart from that you can see a line along the bottom of the screen which is the menu, task bar and system tray.

From the right to the left we have:

1. The off button You can log out to get back to the text prompt, or shut down or reboot the computer.

2. The screen lock button which can be used to lock the screen and keyboard.

3. The clock

4. The CPU monitor which is the green graph. This will spend a lot of time at 100%, but thanks to the power of the Linux kernel the raspberry pi will still feel responsive.

5. Further left filling the main area is a space for icons of running applications.

6. Next is the virtual desktop switcher. This is a really nice feature which lets you have one group of programs running on one virtual desktop and another group on another. You can switch between them at a click. I tend to have a web browser on one and four terminals on the other which I use for development. You can arrange them however you like.

7. Next is the show desktop button, which minimises all the open windows.

8. The next two icons are launchers for frequently used applications. This is the midori web browser.

9. This is the launcher for the file manager.

10. At the far left is the menu itself.

This can be modified as you wish. Right click on the bar for a menu.

I expected to see the python development environment idle in the menu of applications, but it was not there. Instead SPE is installed which looks similar and I assume has more features. Other programming environments are scratch which looks good for getting kids started in programming and squeak which I was not able to get started. Leaf pad is a lightweight editor if you prefer that to using an integrated development environment.

To update the system open a terminal window, and type:


sudo aptitude


This brings up a program which can be used to manage software. Type u to update the package list. You will see it download some files, then it will read the package lists and go back to the main screen. Then type U (Upper case this time) to mark the packages that can be upgraded. Lastly type g to actually upgrade.

So looking around the system, not many programs are installed. The brilliant thing about debian is that they have packaged a massive amount of software which is really easy to install. You can use aptiutude to search – use CTRL+T to bring up the menu. You can search for packages, mark them for install and install them. If you want a graphical program to administer it, I recommend using synaptic. To install it, open a terminal and type:

sudo aptitude install synaptic

After some messages scroll past the screen and it checks if you really want to install the software, it is installed. Then you can find the program in the menu, run it and look through all the software that can be installed.

Alternatively you can use the command line. It is easier than using the GUI really. The aptutude manual is on the debian wiki. So to upgrade, you do:


sudo aptitude update
sudp aptitude safe-upgrade


To search for a program you can use


apt-cache search minesweeper


And to install a program you can use


sudo aptitude install xdemineur


The program will tell you if it is installing any other dependencies (Libraries and programs) and if so will ask for confirmation. It will say how much space is expected to be used, and how much data will be downloaded. Who needs a graphical interface!

Next time I will post a few thoughts on the Raspberry Pi, and then I will go back to the introduction to python.

Raspberry Pi Python introduction part 2

So, the last tutorial was very nice, but not very pretty. How do we get some graphics?

Here is a really simple python graphical demo. I hope you get some enjoyment from typing it in, making it run, and changing it to make it better. The idea is to get a line wandering around the screen, changing colour. I think it actually looks rather nice!

So, first of all we need to tell python which modules we are going to use. If you don’t know, a module is a collection of program fragments which have already been written. We use these building blocks to make our lives much simpler. If you are writing a program, rather than writing your own physics simulator, statistics package, music player or whatever, you should check and see if anyone has written a module rather than reinvent the wheel.

In this program we will need pygame for the graphics, sys for the operating system code to exit the program, and random to get random numbers.

When you start pygame, you have to run pygame.init() to make it work properly, so I do that here.


import pygame, random, sys
pygame.init()


Next create some variables to store the size of the window the program will create. I have picked a 600 pixel square window. I do it this way so that if you have a different size screen you can easily change the program to cope with the new size. I use these variables to set two new variables, x and y, for the starting point of the line. I put it in the middle of the window.


sizeX=600
sizeY=600

x=sizeX/2
y=sizeY/2


Here is the first time we use pygame. We create a colour. In pygame a colour is an object. An object is like a supercharged variable. It actually contains a number of variables, and methods, which is code you can call to control the object. We will be doing this later, but all we are doing now is calling the constructor which creates the object and puts it in the colour variable.

The colour does not really matter, it just needs to be set to something to start. I picked gray. You could pick any of the allowed colours which are listed in the x11 colour names article at wikipedia.


colour = pygame.Colour('Gray')


[Edit – This does not work in debian – I get:


Traceback (most recent call last):
File "", line 1, in
ValueError: invalid color name


It looks like you can’t use colour names. Instead, use:


colour = pygame.Colour(#444444')


I will also update the program below.

Edit end]

Next we use pygame to create the window the size we decided earlier. The window is another object. We need to assign it to a variable so we can draw on it later. Next we set the title on top of the window.


window = pygame.display.set_mode([sizeX,sizeY])
pygame.display.set_caption("Random art")


Now we have everything ready. We have a colour, a starting point and a window to draw in. Now we can start drawing. We do this in a loop, we want to keep running the same instructions over and over again. We use a while loop, which keeps on executing while the expression is true. Since True is always true, the loop will keep going forever unless we make another way to stop it.


while True:


Now, everything in the loop needs to be indented, and by the same amount, so that python knows to keep it in the loop. If you forget to indent a line python will think it is outside the loop, so it will run once the loop has finished (Which is never in this case). I have chosen to indent by four spaces. I never use tabs, because 8 spaces looks the same as 1 tab, but it is different to python, so it can get confusing.

The first thing to do in the loop is to pick a colour. The nice thing about the colour object is that there are several ways of manipulating it. I like thinking about a colour as a mixture of the three primary colours of light – red, green and blue. I want to choose a colour that looks similar to the last one. So I add or subtract a number between 0-3. I use randint from the random module which we imported earlier to choose a number between -3 and 3, and add it to the previous brightness. The problem is that the brightness can only be between 0 and 255. We can keep it below 255 by dividing the number by 255 and taking the remainder. This is what the %255 bit does. I do this three times, once for each of the primary colours.

    colour.r = (colour.r+random.randint(-3,3)) % 255
    colour.g = (colour.g+random.randint(-3,3)) % 255
    colour.b = (colour.b+random.randint(-3,3)) % 255

Next, pick a new point to draw to. We can do the same thing here as we did with the colours, just add a random number to the point. the new point would not appear on the window, I will just move it to the closest point on the window, otherwise the line might go wandering off past the edge of the window and we would never see it again.

    newx=x+random.randint(-3,3)
    newy=y+random.randint(-3,3)

    if newx > sizeX:
       newx = sizeX
    if newy > sizeY:
       newy = sizeY
    if newx < 0:
       newx = 0
    if newy < 0:
       newy = 0

If I used the same trick as I did with the colours, we would get a line across the screen where it goes from the window size to zero. Why don’t you try it? In that case we could remember this had happened and not draw the line on that occasion. but that is for you to try. What I need to do next is actually draw the line. We use pygame for that.

    pygame.draw.line(window,colour,(x,y),(newx,newy),2)
    pygame.display.flip()

In the first line we draw a line on to our window, using the colour we picked earlier, starting from point (x,y) and ending at point (newx,newy) and we make the line 2 pixels thick. You can experiment with the line thickness. You can read the documentation for pygame.draw on the pygame site. What is the display.flip? The thing is that we didn’t really draw the line on the screen. We drew it in memory and flipping the display copies the contents of the memory to the screen. This will be useful for more complex programs because it means we can make sure we have drawn exactly what we want before putting it on the screen. Also, writing to the screen is a complicated business. The computer has to do the work at the right instant to make sure it is not drawing during a refresh, so only half the picture gets on the screen. Again the pygame site tells you all about the display object.

The next time the loop is run, I want the line to carry on from the end of the line that was just drawn. To do that we have to tell the computer to do that, and I do it like this:

    x=newx
    y=newy

That is everything! If you run the program it will work now! There are a number of problems with it though. The main one is that we have not provided a way to quit the program. Remember the loop will run for ever unless we arrange another way to break out of it, and we have not. Lets add one now.

    for event in pygame.event.get():
        if event.type == pygame.QUIT:
            sys.exit()

So, what this does is tells pygame to look at the events it has received. We will talk more about events in a later tutorial, but an event is basically anything that happens on the computer, like the mouse moving, a button clicking or keys being pressed on the keyboard. We only look at events we are interested in. This event is pygame.QUIT, which means the operating system is telling us that it wants the program to quit. This happens if the user clicks the X on the top corner of the window. So, if the user clicks X, we want to exit. This is where we use exit which is provided by the sys module, which we imported at the top.

Next time we will do another similar program, but address some of the problems in this program, and add a few more features such as some simple user interaction.

Please use the comments box below if you spot any mistakes in the tutorial. If you are using this to learn python, then please let me know how you are getting on, and whether it was helpful.

Last of all, here is the full program:

import pygame, random, sys

# You have to call this to make it work.
pygame.init()

# Set up some variables containing the screeen size
sizeX=600
sizeY=600


# Start drawing from the middle of the window.
x=sizeX/2
y=sizeY/2

# Set the starting colour
colour = pygame.Color('#444444')

# Create the pygame window
window = pygame.display.set_mode([sizeX,sizeY])

# Put a name on the window.
pygame.display.set_caption("Random art")

# This will loop forever.
while True:
    colour.r = (colour.r+random.randint(-3,3)) % 255
    colour.g = (colour.g+random.randint(-3,3)) % 255
    colour.b = (colour.b+random.randint(-3,3)) % 255

    newx=x+random.randint(-3,3)
    newy=y+random.randint(-3,3)

   # Make sure the new point is visible
    if newx > sizeX:
       newx = sizeX
    if newy > sizeY:
       newy = sizeY
    if newx < 0:
       newx = 0
    if newy < 0:
       newy = 0

    pygame.draw.line(window,colour,(x,y),(newx,newy),2)
    x=newx
    y=newy
    pygame.display.flip()
    for event in pygame.event.get():
        if event.type == pygame.QUIT:
            sys.exit()

Raspberry Pi – Python introduction

When I was small, my dad borrowed a BBC computer. and I started typing in programs from books and magazines. I had loads of fun making them work. Some of them were even fun to play!

The people at the Raspberry Pi foundation are trying to bring the same joy to children now. I bought one. I don’t have it yet, but I am eager to get started.

Some people criticise the computer saying it is difficult. When I started the BBC computer, I got the following message:


Acorn MOS
Acorn 1770 DFS
BASIC
>


You had no idea what to do unless you read the manual. I spent ages reading the manual and trying things out. It is similar with the Raspberry Pi

Once you plug in your computer and start it up, you can take a look at SPE. Find it in the menu and run it.

At the bottom of the screen is a message that looks a bit like the message on the BBC. What do you do? When I was at school the first thing we did was write a program which said hello. Over and over! Why not try it now? Type in to the prompt with three chevrons:


print("Hello")


It says Hello!

We have called a python function which takes the argument it is given (“Hello”) and puts it on the screen.

Now try this:

while True:
    print("Hello")

When you press return again, it prints hello over and over again. To stop it you can hold down the Ctrl key and hit C. Or you can restart the shell using “Restart Shell” from the “Shell” menu.

What we have done here is created a loop which checks the value of an expression. If the expression evaluates to true, the loop is run again. Otherwise it goes on to the next instruction after the loop. But in this case, we have set the expression to true, which of course will always be true. So the loop will run for ever unless we do something else to stop it, which we did by hitting control-C, or by restarting the shell.

Here is another program you can type in. This is almost a fun game. Create a new window to type the code in (File->New Window)

import random

secret=random.randint(1,10)
guesses=0
 
while guesses < 10:
    guesses = guesses + 1
    print("I am thinking of a number between 1 and 10.")
    guess=int(input("What is your guess? "))
    if guess > secret:
        print("Too high!")
    elif guess < secret:
        print("Too low")
    else:
        print("You got it! Hooray")
        break

Now save the code to a file (file->save – Pick a file name to save it to – maybe guess.py). Run it using Run-&gtRun module. If all goes well you should get a number guessing program.

This introduces a number of aspects of python. Let’s have a look at what each line is doing.


import random


We need to use some code that python supplies to generate random numbers. This tells python that we want to use this code.


secret=random.randint(1,10)


This uses the supplied function to generate a random number between 1 and 10.


guesses=0


So that the program does not go on for ever, we want to get it to count the number of guesses the user makes. This sets the number of guesses to zero, because nobody has made a guess yet.


while guesses < 10:


We recognise this as a loop. This time it will loop while the number stored in the guesses variable is less than 10.

    guesses = guesses + 1

Python knows what is in the loop because it is indented. Everything that is indented will be in the loop, and if a command is not indented python knows it is not in the loop. It provides a nice way that we can tell at a glance what is in the loop. This line simply adds one to the number of guesses. Remember when it gets to 10, the loop will end.

    print("I am thinking of a number between 1 and 10.")
    guess=int(input("What is your guess? "))

We already know what print does. The next line uses the input function. This prints a message on the screen and takes a line of text from the keyboard. Python does not know what the user will enter, so it stores it in a text string. We want a number, so we convert the text string to an integer using the int function. Then we store the result in the guess variable.

    if guess > secret:
        print("Too high!")

This line introduces the if statement. Like the while loop it controls the flow of the program using a condition statement. If the statement is true the block of code at the next indent level is run. In this case the message is printed that the level is too high.

    elif guess < secret:
        print("Too low")

Here is another part of the if statement. “elif” means else if. It tells python that if the first part is not true, try this next part. There can be lots of elif sections in the if statement.

    else:
        print("You got it! Hooray")

At the end of all the elif sections, you can have an else section. We know that if the number entered was not larger or smaller than the secret number, it must be the same, so we tell the user so.

        break

Notice the indent is still the same as the previous line, so this is still part of the block of code run if the number was guessed correctly. It breaks out of the loop, there is no point in asking for another guess, the user has already solved the problem.

To learn more about python read the manual! There is an introduction to python which will give much more information. On the same page, section 3.2 gives more detail on the statements I have introduced.

Well, that is episode 1. Please let me know what you think, especially if you have used it to get started programming, you spot an error in what I say, or you can think of a way to improve it.