Prepare
Enviroment
拉取docker hub
中ubuntu20.04
镜像,并配置其中的环境(使用docker主要是为了可移植性吧,并且试了一下,从头搭建一个干净环境比在主机上运行,效率高不少。因为在后面已经做好了映像文件,所以配置部分不太重要,有点琐碎,之后再整理dokerfile)
:
$ docker pull ubuntu:20.04
$ docker run -it --name ubuntu_pytorch ubuntu:20.04
之后就可以使用命令来打开并进入容器:
$ docker start ubuntu-pytorch
$ docker exec -it ubuntu-pytorch /bin/bash
安装好必要的工具:
$ apt-get update
$ apt-get install gcc
$ apt-get install wget
$ apt-get install make
$ apt-get install vim
$ apt-get install zlib*
$ apt-get install libffi-dev
$ apt-get install libssl-dev
$ apt-get install python3
$ apt install python3-pip
安装好以后,因为是以nano GPT
的使用为导向,所以根据其文档,来安装对应版本的库
首先要对pip3
换源(原本的源太折磨了)
$ pip3 install --upgrade pip
$ pip3 config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
接着安装其它需要的库(在此之前,需要创建一个普通用户,否则无法正常使用pip安装)
$ pip3 install tqdm==2.2.3
$ pip3 install wandb==0.6.31
$ pip3 install tiktoken==0.3.3
$ pip3 install datasets==2.11.0
$ pip3 install transformers==2.9.0
$ pip3 install numpy==1.24.2
后面就是,比较麻烦的部分:
因为显卡资源在主机上,所以无法在docker
中直接被调用,如果想要使用,就不像在宿主机中,直接安装驱动那么简单,需要在主机上,安装适用于docker
的驱动
本来到了这一步,已经安装了只适用于CPU
版本的pytorch
,打算凑合用了,或者直接使用已经封装好的,还有CUDA
的镜像,后来发现了一个,在WSL
中,透传使用显卡
于是在docker
中查看了一下ubuntu
的版本,发现也是基于WSL
的:
$ uname -r
5.10.102.1-microsoft-standard-WSL2
决定尝试安装对应的驱动,在docker
中,透传使用GPU
先在官网中,找到适用于WSL
的驱动程序,发现Nvidia for WSL2
已经下架了,反而在最新的驱动程序中,描述其适用于WSL
,所以在主机上安装好对应的驱动(操作较为简单,并且在不同主机,版本不同,这里就不加赘述)
在WSL
中,官方文档的描述是,下载安装新的驱动,会覆盖掉WSL
中的CUDA
库,所以需要另外安装官方提供的CUDA
工具包:
$ wget https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/cuda-keyring_1.0-1_all.deb
$ sudo dpkg -i cuda-keyring_1.0-1_all.deb
$ sudo apt-get update
$ sudo apt-get -y install cuda
使用docker
保存好镜像后,重新使用其GPU
功能(docker nvidia的替代)
,创建新的容器:
$ docker run -it --name env_test --gpus all d1msh1mm32/nanogpt:wsl
$ docker exec -it env_test /bin/bash
$ nvidia-smi
再安装GPU
版本的pytorch:
$ sudo pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
最后检查CUDA
是否可以被成功调用:
# test.py
import torch
print(torch.__version__)
print(torch.cuda.is_available())
print(torch.cuda.device_count())
print(torch.version.cuda)
$ python test.py
成功安装pytorch
,并且可以正常调用Nvidia
显卡:Get force 2060T
最后将更新后的容器,封装为新的image
,上传到私有docker hub
中(d1msh1mm32/nanogpt:1.0)
该版本,仅仅是配置好环境,未开始训练模型的映像,所以相对较小
Trial
配置好环境后,拉取了nanoGPT
的代码,运行之前,因为自己的GPU
版本较老,并不支持新的pytorch
下的bfloat16
数据类型,将所有用到的地方都换成了float16
,并且为了提高训练效率,增大了batch_size
,并且在单GPU
的条件下,关闭了compile
功能
使用其中的莎士比亚诗歌的数据集,尝试训练模型:
python3 data/shakespeare_char/prepare.py
python3 train.py config/train_shakespeare_char.py --batch_size=32 --compile=False
最后训练结束后,一共迭代了2500
次,大概用了4h
,loss val
降到了0.28
多一点,查看该模型下,生成的莎士比亚风格的短诗效果:
root@e8f2b2d49e25:/home/d1m/nanoGPT# python3 sample.py --out_dir=out-shakespeare-char
Overriding: out_dir = out-shakespeare-char
number of parameters: 10.65M
Loading meta from data/shakespeare_char/meta.pkl...
ANGELO:
Ay, coward, is not to be so?
ISABELLA:
Good night?
ISABELLA:
For I will be there into a full of my face,
And by my babe some further of my head
Which misery I have laid my drevotion, in me now
With a most prince, and thus wondrous preparents
With deny move and do I live you:
If we might know the king of it.
ISABELLA:
His is noble consent of his business nurse,
If I am not to show thy fledge,
To fight him how that be burnt our sad advance.
EDWARD:
It is too are no more stoop of that
---------------
Men pardon, I will not so go to hold to myself.
LEONTES:
Well, and thou art not to the court.
LADY ANNE:
I should so, my lords, I love thee see
In all the very mad assure of my knees;
How is the husband is that never may should not have done in,
To make heaven and the world the way soul of him,
And, thou wert not a substand himself?
Therefore, I am not doubt, your grace is in out of the cell.
GLOUCESTER:
Pray, that I, brother, my lord and my love to the crown,
And made thee, for God, king one
---------------
MARIANA:
I beseech you, I would not which you
Which you dissemblance the house of your eyes:
This sister your infinitent of the heart world,
And seem to the sword I have a cause of your voices.
Then to do with the royal leisure witness which yet I will
It is stick for Henry's mother brother than direction make her doth rest,
And this and be long in still be brief on the duke of my first:
Henry cannot be the state of Hereford, and he is none to me,
And teny my letter find yield me to rejoice,
Tha
---------------
Think with justice of place, say 'Tis fill'd with virtue.'
County courteey, come, lords, still be so much.
GLOUCESTER:
The world prince' the mayor are and all the heart:
Why, the court hither of some that his sense confirm'd in the brace
Of his majesty corrupt cold, of monaltinous,
And cause his swords apollable and the coronate in the world,
He is not how to scarce a king-owl and sword, proud him disconsent,
But he be not stander stoop and his brother men's face.
LADY ANNE:
My lord, I am all
---------------
Be every believed him to the crown
And let him brave it it flatter it.
ANGELO:
To bring him to the worst,
In a world of late of this eye,
And bring sorrow of you in this decree.
ESCALUS:
I will please you on my lord:
I love the prepare not more your honour countryment;
And you will take you to acquainty that you would prove our husband.
POMPEY:
Marry, my good mother, the sense to me.
ESCALUS:
No, but, he is in joy the villain.
ESCALUS:
It is the young horn of leaven upon the treasure of him
---------------
MENENIUS:
He hath he stain'd with the bosom of their loves.
BRUTUS:
Come, sir, let me hence; I will together
By your foots with your brother. For your death testing for you.
CORIOLANUS:
What, is the matter?
BRUTUS:
I am, he had done to the rest way,
The senate-house of which, here betwixt you were here:
The valiant Camillo walk again within the prince of her to the crown
And thy corruption with thy side hide and his hand.
Why, since thou the king of head of the depention,
But in the balm of
---------------
She will well have been it well
And then, when in a shepherding to hell.
LEONTES:
For that I come: yea, I know thy villain!
PAULINA:
How now!
They have pasticular have of their ruin and ragen
Of vaults are step their scorns, and not one sounds when they?
PERDITA:
On them, assembly doth and this imprisonment to what I will be born to
How you find attend his trumpet, which I must have from you?
First Musician:
Have you known to have a spirit of the city of Paris?
MENENIUS:
Sweet sleep and you
---------------
I'll play them again.
HORTENSIO:
And so I can do not be it.
POLIXENES:
Not I say not persuade the bowls are gone my neckon nor
spection enough: what I will skill be dull upon yourselves
This shall be you not so in the banishment, so I have last
Into your good shall remain tears to see your compounds, and postern
In after to be the varlet of such birth the hatred
Whose of sour and feeting stroke of death, and both purpose about by our exile.
The poisoner revolted with your hands and such as you
---------------
GLOUCESTER:
Lord Henry, that you was our children like earl.
BENVOLIO:
Overween we can tell thee, sit Romeo.
BALTHASAR:
Not at this sound is Aufidius the mistress of him.
Romeo, I see thither did revolt him that were known
The all true that hath successed continued the cheek of you.
MERCUTIO:
The same of what have I will deput thee to do't.
ROMEO:
Ay, thou hast not set the warrant of this heart
And thou hadst not break our brief. I think that four heart.
ROMEO:
Come, my mouth, and fear it i
---------------
He hath not set drops:
The west had not of that I am here in the regal flower,
And yet make a gap of kingdom coward so have tell me,
Destrius, thou mayst prove entreat thou art
A son of thy wanton when I should not well.
Thou stoop'dst, mighty more shall shall have offend the noble come
Of nought to England's power; and for though I could take him forget
His arms and could be the heavens of my infliction:
For I see him in this mething of his right,
If I be he is such a man makes of him
How my re
---------------
最后可以调用:
python3 data/shakespeare/prepare.py
python3 train.py config/finetune_shakespeare.py --init_from=gpt2
进行fine tune
,其默认使用的初始化模型为gpt2-xl
,该模型太大,只是用gpt2
进行微调
简单迭代了20
次,得到的loss val 3.4002
,再查看生成的文本效果:
root@e8f2b2d49e25:/home/d1m/nanoGPT# python3 sample.py --out_dir=out-shakespeare
Overriding: out_dir = out-shakespeare
number of parameters: 123.65M
No meta.pkl found, assuming GPT-2 encodings...
...
The first attempt was reported as being made on Monday, but reports in the newspapers on Tuesday made it into circulation on Wednesday.
The Polish Air Force was assisting the Russian air force in the attack.
The Polish air defence corps was bombing the air base in the north of the country.
A number of bombs were dropped from the air base, including a rocket with the name Krakow, which was later in use by the Polish army.
The Polish air force used two Soviet-made rockets as well as two to the east of Krakow.
The Russian air defence corps carried out an attempt to cross into Belarus through Poland.
In the northern part of Poland, the Polish army attacked a Russian base in the south half of the country.
The Polish air force's main force, the Polish Expeditionary Force, was launched into the country on the orders of the government.
Poland's main army was carrying out a counter-attack on a Polish ally, and was then met by Soviet-backed forces.
The Polish military was preparing for an attack on the Soviet base at Krakow, with a military force of 750,000, which was to be made up of the Polish army and the Warsaw Pact.
The Polish army's main force was carrying out an air attack on the base at the request of Poland.
The Polish air force was preparing for an attack on the Warsaw Pact's base at Krakow and a counter-attack.
The Polish military was preparing for a counter-attack on the Warsaw Pact's base at Krakow.
The Polish military was preparing for any attempt aimed at crossing into Poland by Poland's main forces.
The Polish army had made an armed attack in the south part of the country.
The Polish army was prepared to attack the Warsaw Pact's base at Krakow at the request of Poland.
The Polish army had made an armed attack in the south part of the country.
The Polish army was prepared to attack the Warsaw Pact's base at Krakow at the request of Poland.
And the Polish army was prepared to attack the Warsaw Pact's base at Krakow.
The Polish army had made an armed attack in the south part of the country.
The Polish soldiers had made an armed attack in Poland.
And the Polish soldiers had made an armed attack in
---------------
SINCE the next project, the most anticipated is the first of the series will be called Aftermath: a comic series about one of the most important issues in the history of mankind, the world of the human race.
The first issue of the series to follow is The Aeon Rising (2003) in which I'm joined by Rick and Laura from the Theresienstadt Library to discuss the work of Steven Soderbergh and their great influence on comics and their return to print this year.
Don't forget to subscribe to The Aeon Rising to get new episodes every week, just as you might by subscribing on your favourite podcast platform. Our customers are here:
Click here to access our official RSS feed.
Like this: Like Loading...<|endoftext|>Rivals International:
We've had several more races, but I've had to make a decision: One race is in too far out of my reach.
I'm tired of my own race.
The next race is on!<|endoftext|>Answered by:
This article was originally published on TheHindu.com.
Shriya Mathews has authored a column for The Hindu.
Yesterday, after being elected to a fourth term in the Congress, Mathews, a columnist for The Hindu, was elected representative of the state.
Today, he has taken a second term as a legislator.
He, too, is now an Lok Sabha member.
It is too early to tell how the future of his political career will unfold.
He is waiting to hear what we can learn from him.
To him
Here is a description of the new chief minister of Uttarakhand, whose appointment is yet to be performed.
As a young man, he had been a wealthy merchant, having bought a house, and then retiring from the business of making a deal. He had been married and had two sons. He was a pupil to a blacksmith, who fell ill with a fever, and in consequence had to abort his education. He was married to Mrs. H. G.G. (B.), and had children.
He was till now a young man, having been only fourteen years old, by the time he was fourteen-years-old when he came to the city of Arunachal Pradesh.
He had been engaged in the construction of a railway that would
---------------
As a first step in implementing a solid design, it's good to know what you know. It can lead to a better product or service.
What Are the Benefits of Designing a Business
The first business in our life need to do something.
Designing a business is a lot like building a school.
In the first year, you're never too young to plan.
We want to do the best that we can to achieve our goal.
This means that we are always looking for new ideas.
What are the Costs of Designing a Business
The first 10 years of every business are made up of thousands of hours of design work.
The work takes on a life of its own.
The work we do takes on far more budget than we can afford.
Some of our biggest costs are:
Designing the system
Designing the market
Designing inventory
Designing service
Designing design materials
Designing marketing
Designing sales
Designing promotion
Designing marketing and sales
Designing customer care
Designing sales
Designing marketing and sales
Designing marketing marketing and sales
Designing sales
Designing marketing marketing and sales
Designing sales is a fast way of doing business.
To me, this means that we're the world's best designers.
How to Begin
After we have worked so hard, we have found the best place to start.
The first step is to create a product or service.
A service is a concept that is of some quality.
A very large concept may be a prototype,
but it's also an idea that we'll need to implement it on.
We have to consider every new idea and create a prototype that will fit within the budget.
There is a good reason that most of these concepts come from third world countries.
For instance, in China, we have a concept that is pretty much illegal.
We've had a lot of people come up with concepts and then sell them on.
A lot of us in China have been sold with nothing,
but these people had very little to do with it.
Why is this?
Because when we try to sell something, we have to try to make up for it.
---------------
"The money we have spent has not only helped reduce the military budget, but it has also had a meaningful impact on our national security."
The UN resolution does not specify specific countries, but says they were added to a list of nations where there is a need for help.
The U.N. resolution also says the U.S. should provide food, shelter and water.
The White House is meeting with Turkish President Recep Tayyip Erdogan, Saudi Arabia's King Salman bin Abdulaziz, Qatar's King Fahd bin Abdulaziz and Bahrain's King Salman bin Abdulaziz at the White House on Sunday.
The two leaders have discussed the situation in Syria.
Tensions have been high in the kingdom over its support for the Assad regime.
Assad rebels clashed with Kurdish fighters on Saturday in an offensive in which they were forced to flee.
The U.S. strongly condemned the fighting in Syria's northern city of Raqqa, saying it was "an act of aggression against the Syrian people."
Mr. Hollande said that the U.S. would continue to work with our allies in the region as well as in the world to put an end to Islamic State of Iraq and the Levant (ISIL) and other terrorist groups.
The French president is due to discuss the situation in Iraq in an open-ended meeting next Friday in Baghdad.
He also spoke to Syria's regional partners, including Turkey and Jordan, who are prepared to help in any way the government can, through an early signing of a peace deal.
Meanwhile, President Trump on Wednesday called for Israel to withdraw from the settlements in the West Bank and East Jerusalem.
The White House statement, issued by US Deputy National Security Adviser H.R. McMaster, said the United States and Israel would continue to provide help to the Palestinian Authority in its struggle to achieve a state of peace with Israel.
Mr. Trump has called for a ceasefire in the West Bank and east Jerusalem, a measure that in the past has been met with skepticism as some view it as a false flag operation by Israel.
He said he believed an Israeli-Palestinian state would not be preferable to two states only.
"We stand ready to help and in all circumstances will continue to be prepared to do so," he said, adding that he had made no formal decision on the issue.
The White House is likely to meet with
中间生成了很多奇奇怪怪的东西,应该是中间步骤出了问题,因为损失值也不大对
d1msh1mm32/nanogpt:2.0
,该版本是在下载完gpt2-xl和gpt2
的映像,并对上述的莎士比亚模型训练完并微调结束后push
的,所以大了好多